[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572622576160-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:17:02 dist-test-slave-whsn krb5kdc[395](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:17:02 dist-test-slave-whsn krb5kdc[395](info): set up 2 sockets
Sep 28 05:17:02 dist-test-slave-whsn krb5kdc[395](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260928 05:17:03.609262 388 mini_kdc.cc:121] Time spent starting KDC: real 0.950s user 0.000s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572622576160-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572622576160-388-0/krb5kdc/test-user.keytab.
Sep 28 05:17:03 dist-test-slave-whsn krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:17:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:17:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 05:17:03.761225 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572622576160-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572622576160-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572622576160-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40061
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40061
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:17:04.075999 408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:17:04.076283 408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:17:04.076426 408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:17:04.083879 408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:17:04.083999 408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:17:04.084072 408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:17:04.084146 408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:17:04.084205 408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:17:04.094381 408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572622576160-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572622576160-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40061
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572622576160-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:17:04.095654 408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:17:04.096951 408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:17:04.107882 414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:04.107849 416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:04.107914 413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:17:04.108659 408 server_base.cc:1034] running on GCE node
I20260928 05:17:04.109364 408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:17:04.110664 408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:17:04.111847 408 hybrid_clock.cc:648] HybridClock initialized: now 1790572624111760 us; error 79 us; skew 500 ppm
I20260928 05:17:04.112155 408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:17:04 dist-test-slave-whsn krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572624, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:17:04.115378 408 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 05:17:04.116683 408 webserver.cc:460] Webserver started at http://127.0.97.62:34655/ using document root <none> and password file <none>
I20260928 05:17:04.117345 408 fs_manager.cc:359] Metadata directory not provided
I20260928 05:17:04.117435 408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:17:04.117702 408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:17:04.120432 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572622576160-388-0/minicluster-data/master-0/data/instance:
uuid: "890e23da94e5492d96518d65d5d425ab"
format_stamp: "Formatted at 2026-09-28 05:17:04 on dist-test-slave-whsn"
server_key: "40a692cedc03290e7fb161ff4d54668c"
server_key_iv: "a2e7ccc85bea10a0faa0e9a7710f909b"
server_key_version: "encryptionkey@0"
I20260928 05:17:04.121136 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572622576160-388-0/minicluster-data/master-0/wal/instance:
uuid: "890e23da94e5492d96518d65d5d425ab"
format_stamp: "Formatted at 2026-09-28 05:17:04 on dist-test-slave-whsn"
server_key: "40a692cedc03290e7fb161ff4d54668c"
server_key_iv: "a2e7ccc85bea10a0faa0e9a7710f909b"
server_key_version: "encryptionkey@0"
I20260928 05:17:04.125864 408 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 05:17:04.129128 423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:04.130333 408 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 05:17:04.130461 408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572622576160-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572622576160-388-0/minicluster-data/master-0/wal
uuid: "890e23da94e5492d96518d65d5d425ab"
format_stamp: "Formatted at 2026-09-28 05:17:04 on dist-test-slave-whsn"
server_key: "40a692cedc03290e7fb161ff4d54668c"
server_key_iv: "a2e7ccc85bea10a0faa0e9a7710f909b"
server_key_version: "encryptionkey@0"
I20260928 05:17:04.130606 408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572622576160-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572622576160-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572622576160-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:17:04.147297 408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:17:04.150986 408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:17:04.151196 408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:17:04.167723 408 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40061
I20260928 05:17:04.167743 474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40061 every 8 connection(s)
I20260928 05:17:04.169047 408 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:17:04.173174 475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:17:04.175607 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 408
I20260928 05:17:04.175923 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572622576160-388-0/minicluster-data/master-0/wal/instance
I20260928 05:17:04.178388 388 external_mini_cluster.cc:1428] Setting key 6a8cb8e4f6290324559b4bd5677e4ca6
I20260928 05:17:04.185653 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 890e23da94e5492d96518d65d5d425ab: Bootstrap starting.
I20260928 05:17:04.188977 475 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 890e23da94e5492d96518d65d5d425ab: Neither blocks nor log segments found. Creating new log.
I20260928 05:17:04.189913 475 log.cc:824] T 00000000000000000000000000000000 P 890e23da94e5492d96518d65d5d425ab: Log is configured to *not* fsync() on all Append() calls
I20260928 05:17:04.192425 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 890e23da94e5492d96518d65d5d425ab: No bootstrap required, opened a new log
I20260928 05:17:04.201855 475 raft_consensus.cc:348] T 00000000000000000000000000000000 P 890e23da94e5492d96518d65d5d425ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "890e23da94e5492d96518d65d5d425ab" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40061 } }
I20260928 05:17:04.202286 475 raft_consensus.cc:374] T 00000000000000000000000000000000 P 890e23da94e5492d96518d65d5d425ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:17:04.202409 475 raft_consensus.cc:729] T 00000000000000000000000000000000 P 890e23da94e5492d96518d65d5d425ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 890e23da94e5492d96518d65d5d425ab, State: Initialized, Role: FOLLOWER
I20260928 05:17:04.203084 475 consensus_queue.cc:260] T 00000000000000000000000000000000 P 890e23da94e5492d96518d65d5d425ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "890e23da94e5492d96518d65d5d425ab" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40061 } }
I20260928 05:17:04.203397 475 raft_consensus.cc:388] T 00000000000000000000000000000000 P 890e23da94e5492d96518d65d5d425ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:17:04.203532 475 raft_consensus.cc:482] T 00000000000000000000000000000000 P 890e23da94e5492d96518d65d5d425ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:17:04.203698 475 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 890e23da94e5492d96518d65d5d425ab [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:17:04.205873 475 raft_consensus.cc:504] T 00000000000000000000000000000000 P 890e23da94e5492d96518d65d5d425ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "890e23da94e5492d96518d65d5d425ab" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40061 } }
I20260928 05:17:04.206324 475 leader_election.cc:302] T 00000000000000000000000000000000 P 890e23da94e5492d96518d65d5d425ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 890e23da94e5492d96518d65d5d425ab; no voters:
I20260928 05:17:04.206909 475 leader_election.cc:288] T 00000000000000000000000000000000 P 890e23da94e5492d96518d65d5d425ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:17:04.207052 480 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 890e23da94e5492d96518d65d5d425ab [term 1 FOLLOWER]: Leader election won for term 1
Sep 28 05:17:04 dist-test-slave-whsn krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:17:04.208132 480 raft_consensus.cc:686] T 00000000000000000000000000000000 P 890e23da94e5492d96518d65d5d425ab [term 1 LEADER]: Becoming Leader. State: Replica: 890e23da94e5492d96518d65d5d425ab, State: Running, Role: LEADER
I20260928 05:17:04.208650 480 consensus_queue.cc:237] T 00000000000000000000000000000000 P 890e23da94e5492d96518d65d5d425ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "890e23da94e5492d96518d65d5d425ab" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40061 } }
I20260928 05:17:04.208943 475 sys_catalog.cc:564] T 00000000000000000000000000000000 P 890e23da94e5492d96518d65d5d425ab [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:17:04.211244 481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 890e23da94e5492d96518d65d5d425ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "890e23da94e5492d96518d65d5d425ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "890e23da94e5492d96518d65d5d425ab" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40061 } } }
I20260928 05:17:04.211755 481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 890e23da94e5492d96518d65d5d425ab [sys.catalog]: This master's current role is: LEADER
I20260928 05:17:04.211347 482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 890e23da94e5492d96518d65d5d425ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 890e23da94e5492d96518d65d5d425ab. Latest consensus state: current_term: 1 leader_uuid: "890e23da94e5492d96518d65d5d425ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "890e23da94e5492d96518d65d5d425ab" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40061 } } }
I20260928 05:17:04.212630 482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 890e23da94e5492d96518d65d5d425ab [sys.catalog]: This master's current role is: LEADER
I20260928 05:17:04.213090 485 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:17:04.222765 477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:04.182688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40916 (local address 127.0.97.62:40061)
0928 05:17:04.183587 (+ 899us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:04.183603 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:04.186231 (+ 2628us) server_negotiation.cc:407] Connection header received
0928 05:17:04.194593 (+ 8362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:04.194618 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:04.195072 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:04.195478 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:04.198332 (+ 2854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:04.200426 (+ 2094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:04.202024 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:04.203031 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:04.203214 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:04.209392 (+ 6178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:04.209403 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:04.209422 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:04.209473 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:04.214753 (+ 5280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:04.215817 (+ 1064us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:04.215826 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:04.215835 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:04.216054 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:04.216608 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:04.216614 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:04.216618 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:04.217310 (+ 692us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:04.220714 (+ 3404us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:04.220996 (+ 282us) server_negotiation.cc:299] Negotiation successful
0928 05:17:04.221426 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647,"thread_start_us":387,"threads_started":1}
I20260928 05:17:04.226713 485 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:17:04.236181 485 catalog_manager.cc:1347] Generated new cluster ID: 968e0aba25864f97a00930880ce65a03
I20260928 05:17:04.236294 485 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:17:04.257615 485 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:17:04.259461 485 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:17:04.279881 485 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 890e23da94e5492d96518d65d5d425ab: Generated new TSK 0
I20260928 05:17:04.280628 485 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572622576160-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572622576160-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260928 05:17:04.323515 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572622576160-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572622576160-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572622576160-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572622576160-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40061
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572622576160-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:17:04.644929 501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:17:04.645205 501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:17:04.645311 501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:17:04.652396 501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:17:04.652901 501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:17:04.653057 501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 05:17:04.662420 501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572622576160-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572622576160-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572622576160-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572622576160-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40061
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572622576160-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:17:04.663537 501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:17:04.664793 501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:17:04.677069 506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:04.677310 509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:04.677336 507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:17:04.678510 501 server_base.cc:1034] running on GCE node
I20260928 05:17:04.679031 501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:17:04.679875 501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:17:04.681057 501 hybrid_clock.cc:648] HybridClock initialized: now 1790572624681017 us; error 32 us; skew 500 ppm
I20260928 05:17:04.681334 501 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:17:04 dist-test-slave-whsn krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572624, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:17:04.687345 501 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260928 05:17:04.688603 501 webserver.cc:460] Webserver started at http://127.0.97.1:39423/ using document root <none> and password file <none>
I20260928 05:17:04.689255 501 fs_manager.cc:359] Metadata directory not provided
I20260928 05:17:04.689345 501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:17:04.689586 501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:17:04.692619 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572622576160-388-0/minicluster-data/ts-0/data/instance:
uuid: "f06de23a95f24547ace0db82af9ad8a5"
format_stamp: "Formatted at 2026-09-28 05:17:04 on dist-test-slave-whsn"
server_key: "edaa99af5ed2287601ea5c9a5dc5c0bd"
server_key_iv: "41b7c60272dac87da1ca57fd725e0255"
server_key_version: "encryptionkey@0"
I20260928 05:17:04.693344 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572622576160-388-0/minicluster-data/ts-0/wal/instance:
uuid: "f06de23a95f24547ace0db82af9ad8a5"
format_stamp: "Formatted at 2026-09-28 05:17:04 on dist-test-slave-whsn"
server_key: "edaa99af5ed2287601ea5c9a5dc5c0bd"
server_key_iv: "41b7c60272dac87da1ca57fd725e0255"
server_key_version: "encryptionkey@0"
I20260928 05:17:04.697834 501 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 05:17:04.700796 516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:04.701982 501 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 05:17:04.702121 501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572622576160-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572622576160-388-0/minicluster-data/ts-0/wal
uuid: "f06de23a95f24547ace0db82af9ad8a5"
format_stamp: "Formatted at 2026-09-28 05:17:04 on dist-test-slave-whsn"
server_key: "edaa99af5ed2287601ea5c9a5dc5c0bd"
server_key_iv: "41b7c60272dac87da1ca57fd725e0255"
server_key_version: "encryptionkey@0"
I20260928 05:17:04.702275 501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572622576160-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572622576160-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572622576160-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:17:04.718317 501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:17:04.721915 501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:17:04.722159 501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:17:04.722997 501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:17:04.724630 501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:17:04.724715 501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:04.724829 501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:17:04.724913 501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260928 05:17:04.755729 501 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33163
I20260928 05:17:04.755754 628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33163 every 8 connection(s)
I20260928 05:17:04.756958 501 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572622576160-388-0/minicluster-data/ts-0/data/info.pb
I20260928 05:17:04.757439 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 501
I20260928 05:17:04.757627 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572622576160-388-0/minicluster-data/ts-0/wal/instance
I20260928 05:17:04.758355 388 external_mini_cluster.cc:1428] Setting key c780b38574f8025c2bc076b077efea97
Sep 28 05:17:04 dist-test-slave-whsn krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572624, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260928 05:17:04.777647 631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:04.759399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44477 (local address 127.0.97.62:40061)
0928 05:17:04.759787 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:04.759790 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:04.761001 (+ 1211us) server_negotiation.cc:407] Connection header received
0928 05:17:04.763443 (+ 2442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:04.763447 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:04.763508 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:04.763602 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:04.765379 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:04.766399 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:04.767830 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:04.768376 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:04.768487 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:04.772216 (+ 3729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:04.772223 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:04.772227 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:04.772267 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:04.774881 (+ 2614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:04.775512 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:04.775516 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:04.775518 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:04.775627 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:04.775987 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:04.775991 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:04.775995 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:04.776300 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:04.776424 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:04.777107 (+ 683us) server_negotiation.cc:299] Negotiation successful
0928 05:17:04.777303 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":204,"threads_started":1}
I20260928 05:17:04.778527 632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:04.760011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40061 (local address 127.0.97.1:44477)
0928 05:17:04.760827 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:04.760901 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:04.763170 (+ 2269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:04.763833 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:04.763843 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:04.764471 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:17:04.765226 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:04.765245 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:04.766517 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:04.766520 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:04.767708 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:04.767719 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:04.768554 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:04.768558 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:04.768739 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:04.769336 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:17:04.769360 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:17:04.771990 (+ 2630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:17:04.775001 (+ 3011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:04.775008 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:04.775020 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:04.775424 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:04.775699 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:04.775702 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:04.775704 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:04.775882 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:04.776422 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:17:04.776430 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:17:04.776741 (+ 311us) client_negotiation.cc:769] Sending connection context
0928 05:17:04.777003 (+ 262us) client_negotiation.cc:240] Negotiation successful
0928 05:17:04.777479 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":302,"threads_started":1}
I20260928 05:17:04.780153 629 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40061
I20260928 05:17:04.780416 629 heartbeater.cc:464] Registering TS with master...
I20260928 05:17:04.781118 629 heartbeater.cc:511] Master 127.0.97.62:40061 requested a full tablet report, sending...
I20260928 05:17:04.783187 440 ts_manager.cc:194] Registered new tserver with Master: f06de23a95f24547ace0db82af9ad8a5 (127.0.97.1:33163)
I20260928 05:17:04.785156 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:44477
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572622576160-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572622576160-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260928 05:17:04.795388 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572622576160-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572622576160-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572622576160-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572622576160-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40061
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572622576160-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:17:05.096989 635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:17:05.097229 635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:17:05.097323 635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:17:05.104094 635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:17:05.104203 635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:17:05.104372 635 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 05:17:05.113912 635 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572622576160-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572622576160-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572622576160-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572622576160-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40061
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572622576160-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:17:05.114976 635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:17:05.116190 635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:17:05.128422 640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:05.128469 641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:05.128681 643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:17:05.129511 635 server_base.cc:1034] running on GCE node
I20260928 05:17:05.130041 635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:17:05.130873 635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:17:05.132056 635 hybrid_clock.cc:648] HybridClock initialized: now 1790572625132005 us; error 43 us; skew 500 ppm
I20260928 05:17:05.132326 635 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:17:05 dist-test-slave-whsn krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572625, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:17:05.135149 635 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260928 05:17:05.136459 635 webserver.cc:460] Webserver started at http://127.0.97.2:38883/ using document root <none> and password file <none>
I20260928 05:17:05.137106 635 fs_manager.cc:359] Metadata directory not provided
I20260928 05:17:05.137210 635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:17:05.137475 635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:17:05.140165 635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572622576160-388-0/minicluster-data/ts-1/data/instance:
uuid: "2eaa6f1c62a344acaf5e60df0efb2950"
format_stamp: "Formatted at 2026-09-28 05:17:05 on dist-test-slave-whsn"
server_key: "55cfd25779b682ceea2749a4478858de"
server_key_iv: "e3d85a001076698e31c16f290025fc38"
server_key_version: "encryptionkey@0"
I20260928 05:17:05.140901 635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572622576160-388-0/minicluster-data/ts-1/wal/instance:
uuid: "2eaa6f1c62a344acaf5e60df0efb2950"
format_stamp: "Formatted at 2026-09-28 05:17:05 on dist-test-slave-whsn"
server_key: "55cfd25779b682ceea2749a4478858de"
server_key_iv: "e3d85a001076698e31c16f290025fc38"
server_key_version: "encryptionkey@0"
I20260928 05:17:05.145498 635 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 05:17:05.149011 650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:05.150189 635 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 05:17:05.150317 635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572622576160-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572622576160-388-0/minicluster-data/ts-1/wal
uuid: "2eaa6f1c62a344acaf5e60df0efb2950"
format_stamp: "Formatted at 2026-09-28 05:17:05 on dist-test-slave-whsn"
server_key: "55cfd25779b682ceea2749a4478858de"
server_key_iv: "e3d85a001076698e31c16f290025fc38"
server_key_version: "encryptionkey@0"
I20260928 05:17:05.150470 635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572622576160-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572622576160-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572622576160-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:17:05.224395 635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:17:05.228322 635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:17:05.228579 635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:17:05.229501 635 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:17:05.230701 635 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:17:05.230772 635 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:05.230880 635 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:17:05.230930 635 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:05.261606 635 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42137
I20260928 05:17:05.261639 762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42137 every 8 connection(s)
I20260928 05:17:05.262742 635 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572622576160-388-0/minicluster-data/ts-1/data/info.pb
I20260928 05:17:05.267148 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 635
I20260928 05:17:05.267297 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572622576160-388-0/minicluster-data/ts-1/wal/instance
I20260928 05:17:05.267686 388 external_mini_cluster.cc:1428] Setting key 7fe5f87d539ca8e4c00d638e6da272f4
Sep 28 05:17:05 dist-test-slave-whsn krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572625, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260928 05:17:05.283181 631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:05.265022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36855 (local address 127.0.97.62:40061)
0928 05:17:05.265212 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:05.265218 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:05.266708 (+ 1490us) server_negotiation.cc:407] Connection header received
0928 05:17:05.268440 (+ 1732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:05.268444 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:05.268508 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:05.268590 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:05.270267 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:05.270972 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:05.273126 (+ 2154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:05.273509 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:05.273597 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:05.276892 (+ 3295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:05.276899 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:05.276904 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:05.276939 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:05.280176 (+ 3237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:05.280916 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:05.280922 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:05.280926 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:05.281129 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:05.281558 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:05.281563 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:05.281566 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:05.281879 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:05.282033 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:05.282671 (+ 638us) server_negotiation.cc:299] Negotiation successful
0928 05:17:05.282903 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 05:17:05.284099 765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:05.265580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40061 (local address 127.0.97.2:36855)
0928 05:17:05.266568 (+ 988us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:05.266604 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:05.268181 (+ 1577us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:05.268829 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:05.268838 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:05.269349 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:17:05.270094 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:05.270119 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:05.271114 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:05.271121 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:05.273001 (+ 1880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:05.273015 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:05.273700 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:05.273708 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:05.273911 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:05.274608 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:17:05.274631 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:17:05.276643 (+ 2012us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:17:05.280320 (+ 3677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:05.280327 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:05.280339 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:05.280756 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:05.281257 (+ 501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:05.281261 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:05.281263 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:05.281435 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:05.281994 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:17:05.281999 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:17:05.282300 (+ 301us) client_negotiation.cc:769] Sending connection context
0928 05:17:05.282532 (+ 232us) client_negotiation.cc:240] Negotiation successful
0928 05:17:05.282949 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":775,"thread_start_us":613,"threads_started":1}
I20260928 05:17:05.285463 763 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40061
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260928 05:17:05.285811 763 heartbeater.cc:464] Registering TS with master...
I20260928 05:17:05.286489 763 heartbeater.cc:511] Master 127.0.97.62:40061 requested a full tablet report, sending...
I20260928 05:17:05.287873 440 ts_manager.cc:194] Registered new tserver with Master: 2eaa6f1c62a344acaf5e60df0efb2950 (127.0.97.2:42137)
I20260928 05:17:05.288609 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:36855
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572622576160-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572622576160-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260928 05:17:05.306097 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572622576160-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572622576160-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572622576160-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572622576160-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40061
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572622576160-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:17:05.611711 768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:17:05.611971 768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:17:05.612071 768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:17:05.619009 768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:17:05.619112 768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:17:05.619220 768 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 05:17:05.628358 768 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572622576160-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572622576160-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572622576160-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572622576160-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40061
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572622576160-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:17:05.629578 768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:17:05.630805 768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:17:05.643774 773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:05.643800 774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:05.643800 776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:17:05.644003 768 server_base.cc:1034] running on GCE node
I20260928 05:17:05.644825 768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:17:05.645751 768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:17:05.646939 768 hybrid_clock.cc:648] HybridClock initialized: now 1790572625646887 us; error 46 us; skew 500 ppm
I20260928 05:17:05.647223 768 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:17:05 dist-test-slave-whsn krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572625, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:17:05.650231 768 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260928 05:17:05.651582 768 webserver.cc:460] Webserver started at http://127.0.97.3:45149/ using document root <none> and password file <none>
I20260928 05:17:05.652201 768 fs_manager.cc:359] Metadata directory not provided
I20260928 05:17:05.652302 768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:17:05.652578 768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:17:05.655464 768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572622576160-388-0/minicluster-data/ts-2/data/instance:
uuid: "82052ab229854334be04712ec2687b2b"
format_stamp: "Formatted at 2026-09-28 05:17:05 on dist-test-slave-whsn"
server_key: "49fe59b58204642daa4893c98336501a"
server_key_iv: "059d2ee229e42c71ad68ba02c706dd45"
server_key_version: "encryptionkey@0"
I20260928 05:17:05.656172 768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572622576160-388-0/minicluster-data/ts-2/wal/instance:
uuid: "82052ab229854334be04712ec2687b2b"
format_stamp: "Formatted at 2026-09-28 05:17:05 on dist-test-slave-whsn"
server_key: "49fe59b58204642daa4893c98336501a"
server_key_iv: "059d2ee229e42c71ad68ba02c706dd45"
server_key_version: "encryptionkey@0"
I20260928 05:17:05.660899 768 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260928 05:17:05.664011 783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:05.665277 768 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260928 05:17:05.665408 768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572622576160-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572622576160-388-0/minicluster-data/ts-2/wal
uuid: "82052ab229854334be04712ec2687b2b"
format_stamp: "Formatted at 2026-09-28 05:17:05 on dist-test-slave-whsn"
server_key: "49fe59b58204642daa4893c98336501a"
server_key_iv: "059d2ee229e42c71ad68ba02c706dd45"
server_key_version: "encryptionkey@0"
I20260928 05:17:05.665572 768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572622576160-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572622576160-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572622576160-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:17:05.679236 768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:17:05.682752 768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:17:05.682991 768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:17:05.683845 768 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:17:05.685075 768 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:17:05.685143 768 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:05.685233 768 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:17:05.685282 768 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:05.716200 768 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46137
I20260928 05:17:05.716238 895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46137 every 8 connection(s)
I20260928 05:17:05.717478 768 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572622576160-388-0/minicluster-data/ts-2/data/info.pb
I20260928 05:17:05.719419 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 768
I20260928 05:17:05.719574 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572622576160-388-0/minicluster-data/ts-2/wal/instance
I20260928 05:17:05.720065 388 external_mini_cluster.cc:1428] Setting key 63d4739fa82e4e078062b9e3a91c7a30
Sep 28 05:17:05 dist-test-slave-whsn krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572625, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:17:05.736672 631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:05.720277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44879 (local address 127.0.97.62:40061)
0928 05:17:05.720429 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:05.720433 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:05.721867 (+ 1434us) server_negotiation.cc:407] Connection header received
0928 05:17:05.723958 (+ 2091us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:05.723962 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:05.724023 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:05.724109 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:05.725506 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:05.726312 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:05.727760 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:05.728201 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:05.728298 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:05.731442 (+ 3144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:05.731446 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:05.731449 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:05.731477 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:05.733689 (+ 2212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:05.734426 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:05.734431 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:05.734434 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:05.734567 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:05.735072 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:05.735076 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:05.735078 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:05.735347 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:05.735470 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:05.736128 (+ 658us) server_negotiation.cc:299] Negotiation successful
0928 05:17:05.736332 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 05:17:05.737633 898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:05.720969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40061 (local address 127.0.97.3:44879)
0928 05:17:05.721700 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:05.721759 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:05.723729 (+ 1970us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:05.724325 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:05.724334 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:05.724820 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:17:05.725377 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:05.725394 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:05.726426 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:05.726429 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:05.727626 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:05.727637 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:05.728385 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:05.728390 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:05.728645 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:05.729298 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:17:05.729320 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:17:05.731277 (+ 1957us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:17:05.733823 (+ 2546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:05.733831 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:05.733845 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:05.734306 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:05.734677 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:05.734682 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:05.734686 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:05.734916 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:05.735463 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:17:05.735471 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:17:05.735774 (+ 303us) client_negotiation.cc:769] Sending connection context
0928 05:17:05.736005 (+ 231us) client_negotiation.cc:240] Negotiation successful
0928 05:17:05.736491 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":275,"threads_started":1}
I20260928 05:17:05.738832 896 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40061
I20260928 05:17:05.739127 896 heartbeater.cc:464] Registering TS with master...
I20260928 05:17:05.740198 896 heartbeater.cc:511] Master 127.0.97.62:40061 requested a full tablet report, sending...
I20260928 05:17:05.741479 440 ts_manager.cc:194] Registered new tserver with Master: 82052ab229854334be04712ec2687b2b (127.0.97.3:46137)
I20260928 05:17:05.742259 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:44879
I20260928 05:17:05.747539 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 05:17:05 dist-test-slave-whsn krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572625, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 05:17:05 dist-test-slave-whsn krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572625, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260928 05:17:05.787860 629 heartbeater.cc:503] Master 127.0.97.62:40061 was elected leader, sending a full tablet report...
I20260928 05:17:05.788990 902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:05.774636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35904 (local address 127.0.97.1:33163)
0928 05:17:05.775133 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:05.775141 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:05.775553 (+ 412us) server_negotiation.cc:407] Connection header received
0928 05:17:05.775714 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:05.775721 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:05.775927 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:05.776047 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:05.777323 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:05.778289 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:05.779525 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:05.780055 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:05.780155 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:05.784095 (+ 3940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:05.784105 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:05.784119 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:05.784147 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:05.786553 (+ 2406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:05.787095 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:05.787102 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:05.787104 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:05.787206 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:05.787551 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:05.787554 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:05.787556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:05.787836 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:05.788109 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:05.788411 (+ 302us) server_negotiation.cc:299] Negotiation successful
0928 05:17:05.788618 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":225,"threads_started":1}
W20260928 05:17:05.790222 544 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35904
Sep 28 05:17:05 dist-test-slave-whsn krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572625, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 05:17:05 dist-test-slave-whsn krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572625, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260928 05:17:05.826473 902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:05.814747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35912 (local address 127.0.97.1:33163)
0928 05:17:05.814965 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:05.814972 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:05.815124 (+ 152us) server_negotiation.cc:407] Connection header received
0928 05:17:05.815265 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:05.815269 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:05.815332 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:05.815411 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:05.816530 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:05.817249 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:05.818337 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:05.818700 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:05.818783 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:05.822038 (+ 3255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:05.822043 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:05.822046 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:05.822074 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:05.824586 (+ 2512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:05.825137 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:05.825140 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:05.825142 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:05.825245 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:05.825497 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:05.825500 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:05.825502 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:05.825706 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:05.825863 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:05.826051 (+ 188us) server_negotiation.cc:299] Negotiation successful
0928 05:17:05.826200 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260928 05:17:05.827792 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 501
I20260928 05:17:05.838443 902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:05.828579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35924 (local address 127.0.97.1:33163)
0928 05:17:05.828727 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:05.828730 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:05.829074 (+ 344us) server_negotiation.cc:407] Connection header received
0928 05:17:05.829163 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:05.829167 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:05.829214 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:05.829281 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:05.830294 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:05.830944 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:05.831996 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:05.832390 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:05.832470 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:05.834421 (+ 1951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:05.834426 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:05.834429 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:05.834454 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:05.836479 (+ 2025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:05.837019 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:05.837023 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:05.837025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:05.837149 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:05.837436 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:05.837439 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:05.837441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:05.837648 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:05.837792 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:05.838070 (+ 278us) server_negotiation.cc:299] Negotiation successful
0928 05:17:05.838208 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 05:17:05.839011 624 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:17:05.918203 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 501
I20260928 05:17:05.931079 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 635
Sep 28 05:17:05 dist-test-slave-whsn krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572625, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260928 05:17:05.944633 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:05.931874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57342 (local address 127.0.97.2:42137)
0928 05:17:05.932379 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:05.932386 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:05.932409 (+ 23us) server_negotiation.cc:407] Connection header received
0928 05:17:05.932465 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:05.932471 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:05.932674 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:05.932794 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:05.933855 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:05.934852 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:05.935998 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:05.936497 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:05.936585 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:05.940121 (+ 3536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:05.940131 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:05.940144 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:05.940172 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:05.942533 (+ 2361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:05.943028 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:05.943034 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:05.943036 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:05.943143 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:05.943424 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:05.943427 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:05.943429 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:05.943712 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:05.943919 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:05.944099 (+ 180us) server_negotiation.cc:299] Negotiation successful
0928 05:17:05.944279 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":275,"threads_started":1}
I20260928 05:17:05.945822 758 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:17:06.024166 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 635
I20260928 05:17:06.041894 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 768
Sep 28 05:17:06 dist-test-slave-whsn krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572625, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260928 05:17:06.056293 912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:06.042833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53690 (local address 127.0.97.3:46137)
0928 05:17:06.043388 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:06.043394 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:06.043421 (+ 27us) server_negotiation.cc:407] Connection header received
0928 05:17:06.043478 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:06.043484 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:06.043694 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:06.043840 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:06.045072 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:06.046053 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:06.047274 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:06.047816 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:06.047925 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:06.051653 (+ 3728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:06.051663 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:06.051677 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:06.051708 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:06.054156 (+ 2448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:06.054723 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:06.054729 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:06.054731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:06.054845 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:06.055103 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:06.055106 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:06.055108 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:06.055398 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:06.055613 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:06.055821 (+ 208us) server_negotiation.cc:299] Negotiation successful
0928 05:17:06.056002 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":278,"threads_started":1}
I20260928 05:17:06.057576 891 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:17:06.137666 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 768
I20260928 05:17:06.150314 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 408
I20260928 05:17:06.151353 470 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:17:06.249017 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 408
2026-09-28T05:17:06Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3635 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572622576160-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:17:06 dist-test-slave-whsn krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:17:06 dist-test-slave-whsn krb5kdc[917](info): set up 2 sockets
Sep 28 05:17:06 dist-test-slave-whsn krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260928 05:17:07.216426 388 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.000s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572622576160-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572622576160-388-0/krb5kdc/test-user.keytab.
Sep 28 05:17:07 dist-test-slave-whsn krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572627, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:17:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:17:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 05:17:07.367693 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572622576160-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572622576160-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572622576160-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35377
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35377
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:17:07.661561 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:17:07.661844 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:17:07.661967 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:17:07.668888 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:17:07.668994 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:17:07.669070 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:17:07.669145 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:17:07.669205 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:17:07.678424 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572622576160-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572622576160-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35377
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572622576160-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:17:07.679621 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:17:07.680876 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:17:07.691639 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:07.691668 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:07.691601 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:17:07.692364 930 server_base.cc:1034] running on GCE node
I20260928 05:17:07.693105 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:17:07.694362 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:17:07.695566 930 hybrid_clock.cc:648] HybridClock initialized: now 1790572627695512 us; error 47 us; skew 500 ppm
I20260928 05:17:07.695868 930 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:17:07 dist-test-slave-whsn krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572627, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:17:07.698657 930 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 05:17:07.699987 930 webserver.cc:460] Webserver started at http://127.0.97.62:40859/ using document root <none> and password file <none>
I20260928 05:17:07.700621 930 fs_manager.cc:359] Metadata directory not provided
I20260928 05:17:07.700728 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:17:07.701038 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:17:07.703716 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572622576160-388-0/minicluster-data/master-0/data/instance:
uuid: "b20db27130264168a45c46836cf73ef2"
format_stamp: "Formatted at 2026-09-28 05:17:07 on dist-test-slave-whsn"
server_key: "e89844f2951ccd622ee4e8dc24fbc864"
server_key_iv: "9dce03491760b153222c874816c29353"
server_key_version: "encryptionkey@0"
I20260928 05:17:07.704384 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572622576160-388-0/minicluster-data/master-0/wal/instance:
uuid: "b20db27130264168a45c46836cf73ef2"
format_stamp: "Formatted at 2026-09-28 05:17:07 on dist-test-slave-whsn"
server_key: "e89844f2951ccd622ee4e8dc24fbc864"
server_key_iv: "9dce03491760b153222c874816c29353"
server_key_version: "encryptionkey@0"
I20260928 05:17:07.709100 930 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260928 05:17:07.712184 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:07.713471 930 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:17:07.713630 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572622576160-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572622576160-388-0/minicluster-data/master-0/wal
uuid: "b20db27130264168a45c46836cf73ef2"
format_stamp: "Formatted at 2026-09-28 05:17:07 on dist-test-slave-whsn"
server_key: "e89844f2951ccd622ee4e8dc24fbc864"
server_key_iv: "9dce03491760b153222c874816c29353"
server_key_version: "encryptionkey@0"
I20260928 05:17:07.713788 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572622576160-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572622576160-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572622576160-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:17:07.731717 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:17:07.735399 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:17:07.735638 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:17:07.751205 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35377
I20260928 05:17:07.751232 996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35377 every 8 connection(s)
I20260928 05:17:07.752381 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:17:07.755682 997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:17:07.760635 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 930
I20260928 05:17:07.760842 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572622576160-388-0/minicluster-data/master-0/wal/instance
I20260928 05:17:07.761631 388 external_mini_cluster.cc:1428] Setting key c2b26ed8bf36e74804cec2f60ed1e24e
I20260928 05:17:07.766059 997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b20db27130264168a45c46836cf73ef2: Bootstrap starting.
I20260928 05:17:07.769301 997 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b20db27130264168a45c46836cf73ef2: Neither blocks nor log segments found. Creating new log.
I20260928 05:17:07.770397 997 log.cc:824] T 00000000000000000000000000000000 P b20db27130264168a45c46836cf73ef2: Log is configured to *not* fsync() on all Append() calls
I20260928 05:17:07.773212 997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b20db27130264168a45c46836cf73ef2: No bootstrap required, opened a new log
Sep 28 05:17:07 dist-test-slave-whsn krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572627, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:17:07.781567 997 raft_consensus.cc:348] T 00000000000000000000000000000000 P b20db27130264168a45c46836cf73ef2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b20db27130264168a45c46836cf73ef2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35377 } }
I20260928 05:17:07.782012 997 raft_consensus.cc:374] T 00000000000000000000000000000000 P b20db27130264168a45c46836cf73ef2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:17:07.782130 997 raft_consensus.cc:729] T 00000000000000000000000000000000 P b20db27130264168a45c46836cf73ef2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b20db27130264168a45c46836cf73ef2, State: Initialized, Role: FOLLOWER
I20260928 05:17:07.782706 997 consensus_queue.cc:260] T 00000000000000000000000000000000 P b20db27130264168a45c46836cf73ef2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b20db27130264168a45c46836cf73ef2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35377 } }
I20260928 05:17:07.782925 997 raft_consensus.cc:388] T 00000000000000000000000000000000 P b20db27130264168a45c46836cf73ef2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:17:07.783089 997 raft_consensus.cc:482] T 00000000000000000000000000000000 P b20db27130264168a45c46836cf73ef2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:17:07.783238 997 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b20db27130264168a45c46836cf73ef2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:17:07.783515 1000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:07.763518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55296 (local address 127.0.97.62:35377)
0928 05:17:07.764297 (+ 779us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:07.764308 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:07.764345 (+ 37us) server_negotiation.cc:407] Connection header received
0928 05:17:07.765955 (+ 1610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:07.765977 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:07.766350 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:07.766642 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:07.768228 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:07.769537 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:07.771171 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:07.771920 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:07.772040 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:07.776428 (+ 4388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:07.776435 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:07.776448 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:07.776482 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:07.779381 (+ 2899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:07.780105 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:07.780111 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:07.780117 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:07.780388 (+ 271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:07.780753 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:07.780756 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:07.780758 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:07.781290 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:07.781477 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:07.782000 (+ 523us) server_negotiation.cc:299] Negotiation successful
0928 05:17:07.782373 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":371,"threads_started":1}
I20260928 05:17:07.785595 997 raft_consensus.cc:504] T 00000000000000000000000000000000 P b20db27130264168a45c46836cf73ef2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b20db27130264168a45c46836cf73ef2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35377 } }
I20260928 05:17:07.786108 997 leader_election.cc:302] T 00000000000000000000000000000000 P b20db27130264168a45c46836cf73ef2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b20db27130264168a45c46836cf73ef2; no voters:
I20260928 05:17:07.786580 997 leader_election.cc:288] T 00000000000000000000000000000000 P b20db27130264168a45c46836cf73ef2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:17:07.786695 1002 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b20db27130264168a45c46836cf73ef2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:17:07.787271 1002 raft_consensus.cc:686] T 00000000000000000000000000000000 P b20db27130264168a45c46836cf73ef2 [term 1 LEADER]: Becoming Leader. State: Replica: b20db27130264168a45c46836cf73ef2, State: Running, Role: LEADER
I20260928 05:17:07.787751 1002 consensus_queue.cc:237] T 00000000000000000000000000000000 P b20db27130264168a45c46836cf73ef2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b20db27130264168a45c46836cf73ef2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35377 } }
I20260928 05:17:07.788328 997 sys_catalog.cc:564] T 00000000000000000000000000000000 P b20db27130264168a45c46836cf73ef2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:17:07.790648 1003 sys_catalog.cc:455] T 00000000000000000000000000000000 P b20db27130264168a45c46836cf73ef2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b20db27130264168a45c46836cf73ef2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b20db27130264168a45c46836cf73ef2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35377 } } }
I20260928 05:17:07.791004 1003 sys_catalog.cc:458] T 00000000000000000000000000000000 P b20db27130264168a45c46836cf73ef2 [sys.catalog]: This master's current role is: LEADER
I20260928 05:17:07.791263 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P b20db27130264168a45c46836cf73ef2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b20db27130264168a45c46836cf73ef2. Latest consensus state: current_term: 1 leader_uuid: "b20db27130264168a45c46836cf73ef2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b20db27130264168a45c46836cf73ef2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35377 } } }
I20260928 05:17:07.791594 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P b20db27130264168a45c46836cf73ef2 [sys.catalog]: This master's current role is: LEADER
I20260928 05:17:07.797921 1010 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:17:07.802968 1010 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:17:07.811141 1010 catalog_manager.cc:1347] Generated new cluster ID: 9cb5bb019181498185e735752404aac4
I20260928 05:17:07.811245 1010 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:17:07.838382 1010 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:17:07.839681 1010 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:17:07.849193 1010 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b20db27130264168a45c46836cf73ef2: Generated new TSK 0
I20260928 05:17:07.849960 1010 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572622576160-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572622576160-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260928 05:17:07.894268 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572622576160-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572622576160-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572622576160-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572622576160-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35377
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572622576160-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:17:08.191377 1023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:17:08.191638 1023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:17:08.191773 1023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:17:08.198585 1023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:17:08.198676 1023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:17:08.198824 1023 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 05:17:08.207842 1023 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572622576160-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572622576160-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572622576160-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572622576160-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35377
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572622576160-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:17:08.209069 1023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:17:08.210328 1023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:17:08.221931 1028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:08.221988 1029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:08.222357 1031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:17:08.222644 1023 server_base.cc:1034] running on GCE node
I20260928 05:17:08.223158 1023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:17:08.223954 1023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:17:08.225129 1023 hybrid_clock.cc:648] HybridClock initialized: now 1790572628225079 us; error 42 us; skew 500 ppm
I20260928 05:17:08.225389 1023 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:17:08 dist-test-slave-whsn krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572628, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:17:08.228127 1023 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260928 05:17:08.229401 1023 webserver.cc:460] Webserver started at http://127.0.97.1:40347/ using document root <none> and password file <none>
I20260928 05:17:08.230000 1023 fs_manager.cc:359] Metadata directory not provided
I20260928 05:17:08.230105 1023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:17:08.230366 1023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:17:08.233008 1023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572622576160-388-0/minicluster-data/ts-0/data/instance:
uuid: "96917de6c0944f6bb1062b98fea22b40"
format_stamp: "Formatted at 2026-09-28 05:17:08 on dist-test-slave-whsn"
server_key: "81e7bce41ad025d92e4a8d0671162346"
server_key_iv: "635c0d735322d03b87a779cd056dccca"
server_key_version: "encryptionkey@0"
I20260928 05:17:08.233700 1023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572622576160-388-0/minicluster-data/ts-0/wal/instance:
uuid: "96917de6c0944f6bb1062b98fea22b40"
format_stamp: "Formatted at 2026-09-28 05:17:08 on dist-test-slave-whsn"
server_key: "81e7bce41ad025d92e4a8d0671162346"
server_key_iv: "635c0d735322d03b87a779cd056dccca"
server_key_version: "encryptionkey@0"
I20260928 05:17:08.238216 1023 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260928 05:17:08.241255 1038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:08.242442 1023 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 05:17:08.242583 1023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572622576160-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572622576160-388-0/minicluster-data/ts-0/wal
uuid: "96917de6c0944f6bb1062b98fea22b40"
format_stamp: "Formatted at 2026-09-28 05:17:08 on dist-test-slave-whsn"
server_key: "81e7bce41ad025d92e4a8d0671162346"
server_key_iv: "635c0d735322d03b87a779cd056dccca"
server_key_version: "encryptionkey@0"
I20260928 05:17:08.242749 1023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572622576160-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572622576160-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572622576160-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:17:08.267102 1023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:17:08.270643 1023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:17:08.270888 1023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:17:08.271772 1023 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:17:08.272986 1023 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:17:08.273061 1023 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:08.273172 1023 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:17:08.273221 1023 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:08.303761 1023 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44299
I20260928 05:17:08.303797 1150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44299 every 8 connection(s)
I20260928 05:17:08.305197 1023 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572622576160-388-0/minicluster-data/ts-0/data/info.pb
I20260928 05:17:08.308950 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 1023
I20260928 05:17:08.309152 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572622576160-388-0/minicluster-data/ts-0/wal/instance
I20260928 05:17:08.309878 388 external_mini_cluster.cc:1428] Setting key abcd96ce30fa0ff30460a72c5b3c096c
Sep 28 05:17:08 dist-test-slave-whsn krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572628, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260928 05:17:08.325417 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:08.307448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51463 (local address 127.0.97.62:35377)
0928 05:17:08.307820 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:08.307825 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:08.308823 (+ 998us) server_negotiation.cc:407] Connection header received
0928 05:17:08.311073 (+ 2250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:08.311077 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:08.311136 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:08.311214 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:08.312994 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:08.313679 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:08.315655 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:08.316021 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:08.316140 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:08.319653 (+ 3513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:08.319657 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:08.319661 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:08.319689 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:08.322484 (+ 2795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:08.323179 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:08.323185 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:08.323188 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:08.323323 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:08.323701 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:08.323706 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:08.323710 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:08.324054 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:08.324197 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:08.324830 (+ 633us) server_negotiation.cc:299] Negotiation successful
0928 05:17:08.325062 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":202,"threads_started":1}
I20260928 05:17:08.326247 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:08.307951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35377 (local address 127.0.97.1:51463)
0928 05:17:08.308639 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:08.308689 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:08.310792 (+ 2103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:08.311459 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:08.311467 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:08.312104 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:17:08.312811 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:08.312836 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:08.313823 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:08.313828 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:08.315533 (+ 1705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:08.315547 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:08.316224 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:08.316231 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:08.316441 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:08.317167 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:17:08.317204 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:17:08.319459 (+ 2255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:17:08.322624 (+ 3165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:08.322630 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:08.322641 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:08.323055 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:08.323420 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:08.323424 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:08.323426 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:08.323583 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:08.324181 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:17:08.324187 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:17:08.324470 (+ 283us) client_negotiation.cc:769] Sending connection context
0928 05:17:08.324695 (+ 225us) client_negotiation.cc:240] Negotiation successful
0928 05:17:08.325084 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":257,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260928 05:17:08.327608 1151 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35377
I20260928 05:17:08.327898 1151 heartbeater.cc:464] Registering TS with master...
I20260928 05:17:08.328712 1151 heartbeater.cc:511] Master 127.0.97.62:35377 requested a full tablet report, sending...
I20260928 05:17:08.330753 962 ts_manager.cc:194] Registered new tserver with Master: 96917de6c0944f6bb1062b98fea22b40 (127.0.97.1:44299)
I20260928 05:17:08.332494 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:51463
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572622576160-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572622576160-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260928 05:17:08.345822 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572622576160-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572622576160-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572622576160-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572622576160-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35377
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572622576160-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:17:08.646677 1157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:17:08.646930 1157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:17:08.647027 1157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:17:08.653954 1157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:17:08.654052 1157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:17:08.654210 1157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 05:17:08.663439 1157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572622576160-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572622576160-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572622576160-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572622576160-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35377
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572622576160-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:17:08.664603 1157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:17:08.665858 1157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:17:08.677448 1163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:08.677428 1165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:08.677575 1162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:17:08.678272 1157 server_base.cc:1034] running on GCE node
I20260928 05:17:08.678766 1157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:17:08.679545 1157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:17:08.680728 1157 hybrid_clock.cc:648] HybridClock initialized: now 1790572628680679 us; error 47 us; skew 500 ppm
I20260928 05:17:08.681012 1157 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:17:08 dist-test-slave-whsn krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572628, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:17:08.683811 1157 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260928 05:17:08.685101 1157 webserver.cc:460] Webserver started at http://127.0.97.2:44459/ using document root <none> and password file <none>
I20260928 05:17:08.685704 1157 fs_manager.cc:359] Metadata directory not provided
I20260928 05:17:08.685809 1157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:17:08.686081 1157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:17:08.688890 1157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572622576160-388-0/minicluster-data/ts-1/data/instance:
uuid: "77d62e02a38745cca1634e7f35ddf52c"
format_stamp: "Formatted at 2026-09-28 05:17:08 on dist-test-slave-whsn"
server_key: "0db8b0310e14d8f4f2ade7cd98367cd4"
server_key_iv: "9f84c54433e8c607058ab41583994909"
server_key_version: "encryptionkey@0"
I20260928 05:17:08.689626 1157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572622576160-388-0/minicluster-data/ts-1/wal/instance:
uuid: "77d62e02a38745cca1634e7f35ddf52c"
format_stamp: "Formatted at 2026-09-28 05:17:08 on dist-test-slave-whsn"
server_key: "0db8b0310e14d8f4f2ade7cd98367cd4"
server_key_iv: "9f84c54433e8c607058ab41583994909"
server_key_version: "encryptionkey@0"
I20260928 05:17:08.694370 1157 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260928 05:17:08.697476 1172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:08.698717 1157 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:17:08.698848 1157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572622576160-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572622576160-388-0/minicluster-data/ts-1/wal
uuid: "77d62e02a38745cca1634e7f35ddf52c"
format_stamp: "Formatted at 2026-09-28 05:17:08 on dist-test-slave-whsn"
server_key: "0db8b0310e14d8f4f2ade7cd98367cd4"
server_key_iv: "9f84c54433e8c607058ab41583994909"
server_key_version: "encryptionkey@0"
I20260928 05:17:08.699015 1157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572622576160-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572622576160-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572622576160-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:17:08.720966 1157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:17:08.724769 1157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:17:08.725064 1157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:17:08.725953 1157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:17:08.727129 1157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:17:08.727202 1157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:08.727311 1157 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:17:08.727361 1157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:08.753676 1157 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38353
I20260928 05:17:08.753749 1284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38353 every 8 connection(s)
I20260928 05:17:08.754794 1157 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572622576160-388-0/minicluster-data/ts-1/data/info.pb
I20260928 05:17:08.758489 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 1157
I20260928 05:17:08.758664 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572622576160-388-0/minicluster-data/ts-1/wal/instance
I20260928 05:17:08.759027 388 external_mini_cluster.cc:1428] Setting key 27929a1b243ef2ded887cde7b21c56fe
Sep 28 05:17:08 dist-test-slave-whsn krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572628, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260928 05:17:08.775326 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:08.757211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59583 (local address 127.0.97.62:35377)
0928 05:17:08.757395 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:08.757401 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:08.758522 (+ 1121us) server_negotiation.cc:407] Connection header received
0928 05:17:08.760972 (+ 2450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:08.760978 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:08.761070 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:08.761174 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:08.762923 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:08.763863 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:08.766068 (+ 2205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:08.766463 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:08.766572 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:08.769663 (+ 3091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:08.769667 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:08.769671 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:08.769698 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:08.772481 (+ 2783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:08.773192 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:08.773197 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:08.773201 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:08.773353 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:08.773727 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:08.773732 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:08.773736 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:08.774064 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:08.774205 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:08.774803 (+ 598us) server_negotiation.cc:299] Negotiation successful
0928 05:17:08.775014 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 05:17:08.776064 1287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:08.757650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35377 (local address 127.0.97.2:59583)
0928 05:17:08.758374 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:08.758423 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:08.760707 (+ 2284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:08.761377 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:08.761387 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:08.762021 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:17:08.762747 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:08.762769 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:08.764010 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:08.764016 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:08.765939 (+ 1923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:08.765953 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:08.766684 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:08.766693 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:08.766911 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:08.767480 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:17:08.767501 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:17:08.769461 (+ 1960us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:17:08.772615 (+ 3154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:08.772621 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:08.772632 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:08.773067 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:08.773451 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:08.773454 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:08.773456 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:08.773622 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:08.774169 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:17:08.774175 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:17:08.774442 (+ 267us) client_negotiation.cc:769] Sending connection context
0928 05:17:08.774653 (+ 211us) client_negotiation.cc:240] Negotiation successful
0928 05:17:08.774993 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":265,"threads_started":1}
I20260928 05:17:08.777391 1285 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35377
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260928 05:17:08.777665 1285 heartbeater.cc:464] Registering TS with master...
I20260928 05:17:08.778457 1285 heartbeater.cc:511] Master 127.0.97.62:35377 requested a full tablet report, sending...
I20260928 05:17:08.779876 962 ts_manager.cc:194] Registered new tserver with Master: 77d62e02a38745cca1634e7f35ddf52c (127.0.97.2:38353)
I20260928 05:17:08.780625 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:59583
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572622576160-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572622576160-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260928 05:17:08.796041 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572622576160-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572622576160-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572622576160-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572622576160-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35377
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572622576160-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:17:09.094276 1290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:17:09.094532 1290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:17:09.094630 1290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:17:09.101656 1290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:17:09.101745 1290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:17:09.101864 1290 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 05:17:09.111131 1290 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572622576160-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572622576160-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572622576160-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572622576160-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35377
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572622576160-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:17:09.112283 1290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:17:09.113533 1290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:17:09.125777 1295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:09.125808 1298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:09.125829 1296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:17:09.126530 1290 server_base.cc:1034] running on GCE node
I20260928 05:17:09.127115 1290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:17:09.128010 1290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:17:09.129199 1290 hybrid_clock.cc:648] HybridClock initialized: now 1790572629129148 us; error 38 us; skew 500 ppm
I20260928 05:17:09.129482 1290 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:17:09 dist-test-slave-whsn krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572629, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:17:09.132359 1290 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260928 05:17:09.133706 1290 webserver.cc:460] Webserver started at http://127.0.97.3:42535/ using document root <none> and password file <none>
I20260928 05:17:09.134398 1290 fs_manager.cc:359] Metadata directory not provided
I20260928 05:17:09.134510 1290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:17:09.134792 1290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:17:09.137570 1290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572622576160-388-0/minicluster-data/ts-2/data/instance:
uuid: "4742e9156ba346e4b9fcf79305c4eb44"
format_stamp: "Formatted at 2026-09-28 05:17:09 on dist-test-slave-whsn"
server_key: "25473efa25299ff772688fd817f5f27c"
server_key_iv: "f211763a22c3d5d9fe3b1e35d28bcb88"
server_key_version: "encryptionkey@0"
I20260928 05:17:09.138294 1290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572622576160-388-0/minicluster-data/ts-2/wal/instance:
uuid: "4742e9156ba346e4b9fcf79305c4eb44"
format_stamp: "Formatted at 2026-09-28 05:17:09 on dist-test-slave-whsn"
server_key: "25473efa25299ff772688fd817f5f27c"
server_key_iv: "f211763a22c3d5d9fe3b1e35d28bcb88"
server_key_version: "encryptionkey@0"
I20260928 05:17:09.142930 1290 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 05:17:09.146149 1305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:09.147379 1290 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:17:09.147517 1290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572622576160-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572622576160-388-0/minicluster-data/ts-2/wal
uuid: "4742e9156ba346e4b9fcf79305c4eb44"
format_stamp: "Formatted at 2026-09-28 05:17:09 on dist-test-slave-whsn"
server_key: "25473efa25299ff772688fd817f5f27c"
server_key_iv: "f211763a22c3d5d9fe3b1e35d28bcb88"
server_key_version: "encryptionkey@0"
I20260928 05:17:09.147689 1290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572622576160-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572622576160-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572622576160-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:17:09.171046 1290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:17:09.175084 1290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:17:09.175340 1290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:17:09.176236 1290 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:17:09.177500 1290 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:17:09.177573 1290 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:09.177697 1290 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:17:09.177749 1290 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:09.205632 1290 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45693
I20260928 05:17:09.205691 1417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45693 every 8 connection(s)
I20260928 05:17:09.206790 1290 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572622576160-388-0/minicluster-data/ts-2/data/info.pb
I20260928 05:17:09.208909 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 1290
I20260928 05:17:09.209110 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572622576160-388-0/minicluster-data/ts-2/wal/instance
I20260928 05:17:09.209534 388 external_mini_cluster.cc:1428] Setting key 0f6d14d00f03b5dd5842a5f23ddfd856
Sep 28 05:17:09 dist-test-slave-whsn krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572629, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:17:09.225898 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:09.209175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46531 (local address 127.0.97.62:35377)
0928 05:17:09.209361 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:09.209367 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:09.210590 (+ 1223us) server_negotiation.cc:407] Connection header received
0928 05:17:09.212700 (+ 2110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:09.212705 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:09.212783 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:09.212882 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:09.214899 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:09.215631 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:09.217255 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:09.217689 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:09.217795 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:09.220928 (+ 3133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:09.220933 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:09.220936 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:09.220965 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:09.223168 (+ 2203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:09.223814 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:09.223818 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:09.223820 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:09.223926 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:09.224295 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:09.224300 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:09.224303 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:09.224562 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:09.224700 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:09.225371 (+ 671us) server_negotiation.cc:299] Negotiation successful
0928 05:17:09.225582 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 05:17:09.226783 1420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:09.209720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35377 (local address 127.0.97.3:46531)
0928 05:17:09.210433 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:09.210489 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:09.212471 (+ 1982us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:09.213117 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:09.213125 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:09.213634 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:17:09.214722 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:09.214745 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:09.215778 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:09.215781 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:09.217126 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:09.217138 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:09.217915 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:09.217922 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:09.218137 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:09.218745 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:17:09.218766 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:17:09.220717 (+ 1951us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:17:09.223301 (+ 2584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:09.223310 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:09.223326 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:09.223703 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:09.224026 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:09.224031 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:09.224035 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:09.224197 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:09.224689 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:17:09.224696 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:17:09.225033 (+ 337us) client_negotiation.cc:769] Sending connection context
0928 05:17:09.225265 (+ 232us) client_negotiation.cc:240] Negotiation successful
0928 05:17:09.225748 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":271,"threads_started":1}
I20260928 05:17:09.227993 1418 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35377
I20260928 05:17:09.228242 1418 heartbeater.cc:464] Registering TS with master...
I20260928 05:17:09.228901 1418 heartbeater.cc:511] Master 127.0.97.62:35377 requested a full tablet report, sending...
I20260928 05:17:09.230149 962 ts_manager.cc:194] Registered new tserver with Master: 4742e9156ba346e4b9fcf79305c4eb44 (127.0.97.3:45693)
I20260928 05:17:09.230887 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:46531
I20260928 05:17:09.236179 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 05:17:09.253404 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:09.241867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55308 (local address 127.0.97.62:35377)
0928 05:17:09.242027 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:09.242031 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:09.242511 (+ 480us) server_negotiation.cc:407] Connection header received
0928 05:17:09.242640 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:09.242643 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:09.242698 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:09.242776 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:09.244448 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:09.245189 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:09.246475 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:09.246955 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:09.247047 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:09.249056 (+ 2009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:09.249060 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:09.249063 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:09.249093 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:09.251377 (+ 2284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:09.251965 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:09.251969 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:09.251971 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:09.252074 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:09.252328 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:09.252331 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:09.252333 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:09.252619 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:09.252787 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:09.253038 (+ 251us) server_negotiation.cc:299] Negotiation successful
0928 05:17:09.253184 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 05:17:09.259167 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55308:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 05:17:09.266003 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 05:17:09.297758 1430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:09.276694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38353 (local address 127.0.0.1:48768)
0928 05:17:09.289737 (+ 13043us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:09.289754 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:09.289861 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:09.291098 (+ 1237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:09.291105 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:09.291141 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:17:09.291479 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:09.291491 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:09.293315 (+ 1824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:09.293321 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:09.295478 (+ 2157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:09.295492 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:09.296978 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:09.296984 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:09.297165 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:09.297221 (+ 56us) client_negotiation.cc:769] Sending connection context
0928 05:17:09.297326 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 05:17:09.297461 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12924,"mutex_wait_us":195,"thread_start_us":11525,"threads_started":1}
I20260928 05:17:09.298235 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:09.276695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45693 (local address 127.0.0.1:44386)
0928 05:17:09.288823 (+ 12128us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:09.288887 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:09.289080 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:09.291540 (+ 2460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:09.291544 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:09.291567 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:17:09.291913 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:09.291924 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:09.293997 (+ 2073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:09.294002 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:09.295612 (+ 1610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:09.295627 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:09.297370 (+ 1743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:09.297376 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:09.297540 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:09.297577 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 05:17:09.297666 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 05:17:09.297795 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11873,"mutex_wait_us":23,"thread_start_us":12769,"threads_started":1}
I20260928 05:17:09.299151 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:09.277625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56948 (local address 127.0.97.1:44299)
0928 05:17:09.291957 (+ 14332us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:09.291963 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:09.291984 (+ 21us) server_negotiation.cc:407] Connection header received
0928 05:17:09.292052 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:09.292056 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:09.292206 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:09.292364 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:17:09.293036 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:09.294415 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:09.296724 (+ 2309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:09.298222 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:09.298339 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:09.298521 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:09.298725 (+ 204us) server_negotiation.cc:299] Negotiation successful
0928 05:17:09.298881 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14198,"thread_start_us":14108,"threads_started":1}
I20260928 05:17:09.299197 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:09.277207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44299 (local address 127.0.0.1:56948)
0928 05:17:09.290352 (+ 13145us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:09.290367 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:09.290464 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:09.292374 (+ 1910us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:09.292378 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:09.292400 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:17:09.292678 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:09.292686 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:09.294552 (+ 1866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:09.294558 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:09.296555 (+ 1997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:09.296569 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:09.298394 (+ 1825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:09.298401 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:09.298549 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:09.298579 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 05:17:09.298666 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 05:17:09.298795 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13004,"thread_start_us":13033,"threads_started":1}
I20260928 05:17:09.299333 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:09.277712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44386 (local address 127.0.97.3:45693)
0928 05:17:09.290961 (+ 13249us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:09.290971 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:09.291001 (+ 30us) server_negotiation.cc:407] Connection header received
0928 05:17:09.291091 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:09.291098 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:09.291330 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:09.292104 (+ 774us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:17:09.292347 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:09.293851 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:09.295762 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:09.297168 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:09.298621 (+ 1453us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:09.298805 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:09.298859 (+ 54us) server_negotiation.cc:299] Negotiation successful
0928 05:17:09.299000 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13059,"thread_start_us":13106,"threads_started":1}
I20260928 05:17:09.300493 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:09.276812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48768 (local address 127.0.97.2:38353)
0928 05:17:09.290221 (+ 13409us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:09.290230 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:09.290265 (+ 35us) server_negotiation.cc:407] Connection header received
0928 05:17:09.290368 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:09.290376 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:09.290662 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:09.290847 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:17:09.291652 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:09.293172 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:09.295651 (+ 2479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:09.296779 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:09.298249 (+ 1470us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:09.298450 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:09.298514 (+ 64us) server_negotiation.cc:299] Negotiation successful
0928 05:17:09.299179 (+ 665us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13242,"thread_start_us":13111,"threads_started":1}
I20260928 05:17:09.303175 1353 tablet_service.cc:1467] Processing CreateTablet for tablet 41727d7c0bd94286aa763f8dc974e51d (DEFAULT_TABLE table=test-table [id=85539949533549599c3050d9602c5be2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:17:09.303186 1086 tablet_service.cc:1467] Processing CreateTablet for tablet 41727d7c0bd94286aa763f8dc974e51d (DEFAULT_TABLE table=test-table [id=85539949533549599c3050d9602c5be2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:17:09.303730 1220 tablet_service.cc:1467] Processing CreateTablet for tablet 41727d7c0bd94286aa763f8dc974e51d (DEFAULT_TABLE table=test-table [id=85539949533549599c3050d9602c5be2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:17:09.304612 1353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41727d7c0bd94286aa763f8dc974e51d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:17:09.304636 1086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41727d7c0bd94286aa763f8dc974e51d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:17:09.304960 1220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41727d7c0bd94286aa763f8dc974e51d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:17:09.317744 1436 tablet_bootstrap.cc:492] T 41727d7c0bd94286aa763f8dc974e51d P 77d62e02a38745cca1634e7f35ddf52c: Bootstrap starting.
I20260928 05:17:09.320799 1436 tablet_bootstrap.cc:654] T 41727d7c0bd94286aa763f8dc974e51d P 77d62e02a38745cca1634e7f35ddf52c: Neither blocks nor log segments found. Creating new log.
I20260928 05:17:09.321920 1436 log.cc:824] T 41727d7c0bd94286aa763f8dc974e51d P 77d62e02a38745cca1634e7f35ddf52c: Log is configured to *not* fsync() on all Append() calls
I20260928 05:17:09.324229 1436 tablet_bootstrap.cc:492] T 41727d7c0bd94286aa763f8dc974e51d P 77d62e02a38745cca1634e7f35ddf52c: No bootstrap required, opened a new log
I20260928 05:17:09.324457 1436 ts_tablet_manager.cc:1397] T 41727d7c0bd94286aa763f8dc974e51d P 77d62e02a38745cca1634e7f35ddf52c: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260928 05:17:09.324461 1437 tablet_bootstrap.cc:492] T 41727d7c0bd94286aa763f8dc974e51d P 4742e9156ba346e4b9fcf79305c4eb44: Bootstrap starting.
I20260928 05:17:09.326227 1438 tablet_bootstrap.cc:492] T 41727d7c0bd94286aa763f8dc974e51d P 96917de6c0944f6bb1062b98fea22b40: Bootstrap starting.
I20260928 05:17:09.327617 1437 tablet_bootstrap.cc:654] T 41727d7c0bd94286aa763f8dc974e51d P 4742e9156ba346e4b9fcf79305c4eb44: Neither blocks nor log segments found. Creating new log.
I20260928 05:17:09.328514 1437 log.cc:824] T 41727d7c0bd94286aa763f8dc974e51d P 4742e9156ba346e4b9fcf79305c4eb44: Log is configured to *not* fsync() on all Append() calls
I20260928 05:17:09.329434 1438 tablet_bootstrap.cc:654] T 41727d7c0bd94286aa763f8dc974e51d P 96917de6c0944f6bb1062b98fea22b40: Neither blocks nor log segments found. Creating new log.
I20260928 05:17:09.330583 1438 log.cc:824] T 41727d7c0bd94286aa763f8dc974e51d P 96917de6c0944f6bb1062b98fea22b40: Log is configured to *not* fsync() on all Append() calls
I20260928 05:17:09.331087 1437 tablet_bootstrap.cc:492] T 41727d7c0bd94286aa763f8dc974e51d P 4742e9156ba346e4b9fcf79305c4eb44: No bootstrap required, opened a new log
I20260928 05:17:09.331429 1437 ts_tablet_manager.cc:1397] T 41727d7c0bd94286aa763f8dc974e51d P 4742e9156ba346e4b9fcf79305c4eb44: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.007s
I20260928 05:17:09.333823 1438 tablet_bootstrap.cc:492] T 41727d7c0bd94286aa763f8dc974e51d P 96917de6c0944f6bb1062b98fea22b40: No bootstrap required, opened a new log
I20260928 05:17:09.334113 1438 ts_tablet_manager.cc:1397] T 41727d7c0bd94286aa763f8dc974e51d P 96917de6c0944f6bb1062b98fea22b40: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260928 05:17:09.335069 1151 heartbeater.cc:503] Master 127.0.97.62:35377 was elected leader, sending a full tablet report...
I20260928 05:17:09.334868 1436 raft_consensus.cc:348] T 41727d7c0bd94286aa763f8dc974e51d P 77d62e02a38745cca1634e7f35ddf52c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77d62e02a38745cca1634e7f35ddf52c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38353 } } peers { permanent_uuid: "4742e9156ba346e4b9fcf79305c4eb44" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45693 } } peers { permanent_uuid: "96917de6c0944f6bb1062b98fea22b40" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44299 } }
I20260928 05:17:09.335259 1436 raft_consensus.cc:374] T 41727d7c0bd94286aa763f8dc974e51d P 77d62e02a38745cca1634e7f35ddf52c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:17:09.335347 1436 raft_consensus.cc:729] T 41727d7c0bd94286aa763f8dc974e51d P 77d62e02a38745cca1634e7f35ddf52c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77d62e02a38745cca1634e7f35ddf52c, State: Initialized, Role: FOLLOWER
I20260928 05:17:09.336014 1436 consensus_queue.cc:260] T 41727d7c0bd94286aa763f8dc974e51d P 77d62e02a38745cca1634e7f35ddf52c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77d62e02a38745cca1634e7f35ddf52c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38353 } } peers { permanent_uuid: "4742e9156ba346e4b9fcf79305c4eb44" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45693 } } peers { permanent_uuid: "96917de6c0944f6bb1062b98fea22b40" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44299 } }
I20260928 05:17:09.339959 1437 raft_consensus.cc:348] T 41727d7c0bd94286aa763f8dc974e51d P 4742e9156ba346e4b9fcf79305c4eb44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77d62e02a38745cca1634e7f35ddf52c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38353 } } peers { permanent_uuid: "4742e9156ba346e4b9fcf79305c4eb44" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45693 } } peers { permanent_uuid: "96917de6c0944f6bb1062b98fea22b40" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44299 } }
I20260928 05:17:09.340318 1437 raft_consensus.cc:374] T 41727d7c0bd94286aa763f8dc974e51d P 4742e9156ba346e4b9fcf79305c4eb44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:17:09.340444 1437 raft_consensus.cc:729] T 41727d7c0bd94286aa763f8dc974e51d P 4742e9156ba346e4b9fcf79305c4eb44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4742e9156ba346e4b9fcf79305c4eb44, State: Initialized, Role: FOLLOWER
I20260928 05:17:09.341187 1436 ts_tablet_manager.cc:1428] T 41727d7c0bd94286aa763f8dc974e51d P 77d62e02a38745cca1634e7f35ddf52c: Time spent starting tablet: real 0.017s user 0.009s sys 0.008s
I20260928 05:17:09.341199 1437 consensus_queue.cc:260] T 41727d7c0bd94286aa763f8dc974e51d P 4742e9156ba346e4b9fcf79305c4eb44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77d62e02a38745cca1634e7f35ddf52c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38353 } } peers { permanent_uuid: "4742e9156ba346e4b9fcf79305c4eb44" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45693 } } peers { permanent_uuid: "96917de6c0944f6bb1062b98fea22b40" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44299 } }
I20260928 05:17:09.341675 1285 heartbeater.cc:503] Master 127.0.97.62:35377 was elected leader, sending a full tablet report...
I20260928 05:17:09.342234 1418 heartbeater.cc:503] Master 127.0.97.62:35377 was elected leader, sending a full tablet report...
I20260928 05:17:09.342862 1437 ts_tablet_manager.cc:1428] T 41727d7c0bd94286aa763f8dc974e51d P 4742e9156ba346e4b9fcf79305c4eb44: Time spent starting tablet: real 0.011s user 0.007s sys 0.003s
I20260928 05:17:09.344698 1438 raft_consensus.cc:348] T 41727d7c0bd94286aa763f8dc974e51d P 96917de6c0944f6bb1062b98fea22b40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77d62e02a38745cca1634e7f35ddf52c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38353 } } peers { permanent_uuid: "4742e9156ba346e4b9fcf79305c4eb44" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45693 } } peers { permanent_uuid: "96917de6c0944f6bb1062b98fea22b40" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44299 } }
I20260928 05:17:09.345126 1438 raft_consensus.cc:374] T 41727d7c0bd94286aa763f8dc974e51d P 96917de6c0944f6bb1062b98fea22b40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:17:09.345232 1438 raft_consensus.cc:729] T 41727d7c0bd94286aa763f8dc974e51d P 96917de6c0944f6bb1062b98fea22b40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96917de6c0944f6bb1062b98fea22b40, State: Initialized, Role: FOLLOWER
I20260928 05:17:09.345826 1438 consensus_queue.cc:260] T 41727d7c0bd94286aa763f8dc974e51d P 96917de6c0944f6bb1062b98fea22b40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77d62e02a38745cca1634e7f35ddf52c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38353 } } peers { permanent_uuid: "4742e9156ba346e4b9fcf79305c4eb44" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45693 } } peers { permanent_uuid: "96917de6c0944f6bb1062b98fea22b40" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44299 } }
I20260928 05:17:09.347302 1438 ts_tablet_manager.cc:1428] T 41727d7c0bd94286aa763f8dc974e51d P 96917de6c0944f6bb1062b98fea22b40: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260928 05:17:09.401191 1442 raft_consensus.cc:482] T 41727d7c0bd94286aa763f8dc974e51d P 77d62e02a38745cca1634e7f35ddf52c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:17:09.401481 1442 raft_consensus.cc:504] T 41727d7c0bd94286aa763f8dc974e51d P 77d62e02a38745cca1634e7f35ddf52c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77d62e02a38745cca1634e7f35ddf52c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38353 } } peers { permanent_uuid: "4742e9156ba346e4b9fcf79305c4eb44" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45693 } } peers { permanent_uuid: "96917de6c0944f6bb1062b98fea22b40" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44299 } }
I20260928 05:17:09.403069 1442 leader_election.cc:288] T 41727d7c0bd94286aa763f8dc974e51d P 77d62e02a38745cca1634e7f35ddf52c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4742e9156ba346e4b9fcf79305c4eb44 (127.0.97.3:45693), 96917de6c0944f6bb1062b98fea22b40 (127.0.97.1:44299)
I20260928 05:17:09.411304 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:09.403437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52047 (local address 127.0.97.1:44299)
0928 05:17:09.403638 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:09.403643 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:09.404545 (+ 902us) server_negotiation.cc:407] Connection header received
0928 05:17:09.404711 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:09.404715 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:09.404765 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:09.404828 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:17:09.405798 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:09.406779 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:09.409153 (+ 2374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:09.410319 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:09.410442 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:09.410536 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:09.410940 (+ 404us) server_negotiation.cc:299] Negotiation successful
0928 05:17:09.411071 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 05:17:09.411304 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:09.403430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56867 (local address 127.0.97.3:45693)
0928 05:17:09.403617 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:09.403622 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:09.404198 (+ 576us) server_negotiation.cc:407] Connection header received
0928 05:17:09.404395 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:09.404400 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:09.404459 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:09.404548 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:17:09.405581 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:09.406621 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:09.409158 (+ 2537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:09.410319 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:09.410442 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:09.410539 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:09.410941 (+ 402us) server_negotiation.cc:299] Negotiation successful
0928 05:17:09.411058 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 05:17:09.411492 1446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:09.403701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44299 (local address 127.0.97.2:52047)
0928 05:17:09.404442 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:09.404456 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:09.404577 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:09.404899 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:09.404903 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:09.404933 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:17:09.405670 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:09.405678 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:09.406890 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:09.406895 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:09.409021 (+ 2126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:09.409034 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:09.410535 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:09.410545 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:09.410766 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:09.410824 (+ 58us) client_negotiation.cc:769] Sending connection context
0928 05:17:09.410908 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 05:17:09.411060 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":651,"mutex_wait_us":93,"thread_start_us":459,"threads_started":1}
I20260928 05:17:09.411517 1445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:09.403364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45693 (local address 127.0.97.2:56867)
0928 05:17:09.404094 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:09.404109 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:09.404265 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:09.404582 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:09.404586 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:09.404630 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:17:09.405447 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:09.405464 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:09.406734 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:09.406737 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:09.409022 (+ 2285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:09.409034 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:09.410556 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:09.410561 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:09.410764 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:09.410819 (+ 55us) client_negotiation.cc:769] Sending connection context
0928 05:17:09.410906 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 05:17:09.411056 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"mutex_wait_us":112,"thread_start_us":400,"threads_started":1}
I20260928 05:17:09.412432 1106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41727d7c0bd94286aa763f8dc974e51d" candidate_uuid: "77d62e02a38745cca1634e7f35ddf52c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96917de6c0944f6bb1062b98fea22b40" is_pre_election: true
I20260928 05:17:09.412421 1373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41727d7c0bd94286aa763f8dc974e51d" candidate_uuid: "77d62e02a38745cca1634e7f35ddf52c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4742e9156ba346e4b9fcf79305c4eb44" is_pre_election: true
I20260928 05:17:09.412937 1373 raft_consensus.cc:2454] T 41727d7c0bd94286aa763f8dc974e51d P 4742e9156ba346e4b9fcf79305c4eb44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77d62e02a38745cca1634e7f35ddf52c in term 0.
I20260928 05:17:09.412949 1106 raft_consensus.cc:2454] T 41727d7c0bd94286aa763f8dc974e51d P 96917de6c0944f6bb1062b98fea22b40 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77d62e02a38745cca1634e7f35ddf52c in term 0.
I20260928 05:17:09.413537 1176 leader_election.cc:302] T 41727d7c0bd94286aa763f8dc974e51d P 77d62e02a38745cca1634e7f35ddf52c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77d62e02a38745cca1634e7f35ddf52c, 96917de6c0944f6bb1062b98fea22b40; no voters:
I20260928 05:17:09.413910 1442 raft_consensus.cc:2781] T 41727d7c0bd94286aa763f8dc974e51d P 77d62e02a38745cca1634e7f35ddf52c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 05:17:09.414034 1442 raft_consensus.cc:482] T 41727d7c0bd94286aa763f8dc974e51d P 77d62e02a38745cca1634e7f35ddf52c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 05:17:09.414130 1442 raft_consensus.cc:3037] T 41727d7c0bd94286aa763f8dc974e51d P 77d62e02a38745cca1634e7f35ddf52c [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:17:09.416599 1442 raft_consensus.cc:504] T 41727d7c0bd94286aa763f8dc974e51d P 77d62e02a38745cca1634e7f35ddf52c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77d62e02a38745cca1634e7f35ddf52c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38353 } } peers { permanent_uuid: "4742e9156ba346e4b9fcf79305c4eb44" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45693 } } peers { permanent_uuid: "96917de6c0944f6bb1062b98fea22b40" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44299 } }
I20260928 05:17:09.417137 1442 leader_election.cc:288] T 41727d7c0bd94286aa763f8dc974e51d P 77d62e02a38745cca1634e7f35ddf52c [CANDIDATE]: Term 1 election: Requested vote from peers 4742e9156ba346e4b9fcf79305c4eb44 (127.0.97.3:45693), 96917de6c0944f6bb1062b98fea22b40 (127.0.97.1:44299)
I20260928 05:17:09.417541 1373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41727d7c0bd94286aa763f8dc974e51d" candidate_uuid: "77d62e02a38745cca1634e7f35ddf52c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4742e9156ba346e4b9fcf79305c4eb44"
I20260928 05:17:09.417548 1106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41727d7c0bd94286aa763f8dc974e51d" candidate_uuid: "77d62e02a38745cca1634e7f35ddf52c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96917de6c0944f6bb1062b98fea22b40"
I20260928 05:17:09.417742 1106 raft_consensus.cc:3037] T 41727d7c0bd94286aa763f8dc974e51d P 96917de6c0944f6bb1062b98fea22b40 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:17:09.417738 1373 raft_consensus.cc:3037] T 41727d7c0bd94286aa763f8dc974e51d P 4742e9156ba346e4b9fcf79305c4eb44 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:17:09.419802 1373 raft_consensus.cc:2454] T 41727d7c0bd94286aa763f8dc974e51d P 4742e9156ba346e4b9fcf79305c4eb44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77d62e02a38745cca1634e7f35ddf52c in term 1.
I20260928 05:17:09.419886 1106 raft_consensus.cc:2454] T 41727d7c0bd94286aa763f8dc974e51d P 96917de6c0944f6bb1062b98fea22b40 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77d62e02a38745cca1634e7f35ddf52c in term 1.
I20260928 05:17:09.420127 1173 leader_election.cc:302] T 41727d7c0bd94286aa763f8dc974e51d P 77d62e02a38745cca1634e7f35ddf52c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4742e9156ba346e4b9fcf79305c4eb44, 77d62e02a38745cca1634e7f35ddf52c; no voters:
I20260928 05:17:09.420406 1442 raft_consensus.cc:2781] T 41727d7c0bd94286aa763f8dc974e51d P 77d62e02a38745cca1634e7f35ddf52c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:17:09.420790 1442 raft_consensus.cc:686] T 41727d7c0bd94286aa763f8dc974e51d P 77d62e02a38745cca1634e7f35ddf52c [term 1 LEADER]: Becoming Leader. State: Replica: 77d62e02a38745cca1634e7f35ddf52c, State: Running, Role: LEADER
I20260928 05:17:09.421264 1442 consensus_queue.cc:237] T 41727d7c0bd94286aa763f8dc974e51d P 77d62e02a38745cca1634e7f35ddf52c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77d62e02a38745cca1634e7f35ddf52c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38353 } } peers { permanent_uuid: "4742e9156ba346e4b9fcf79305c4eb44" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45693 } } peers { permanent_uuid: "96917de6c0944f6bb1062b98fea22b40" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44299 } }
I20260928 05:17:09.425446 961 catalog_manager.cc:5495] T 41727d7c0bd94286aa763f8dc974e51d P 77d62e02a38745cca1634e7f35ddf52c reported cstate change: term changed from 0 to 1, leader changed from <none> to 77d62e02a38745cca1634e7f35ddf52c (127.0.97.2). New cstate: current_term: 1 leader_uuid: "77d62e02a38745cca1634e7f35ddf52c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77d62e02a38745cca1634e7f35ddf52c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38353 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4742e9156ba346e4b9fcf79305c4eb44" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96917de6c0944f6bb1062b98fea22b40" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44299 } health_report { overall_health: UNKNOWN } } }
Sep 28 05:17:09 dist-test-slave-whsn krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572627, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
W20260928 05:17:09.458212 1419 tablet.cc:2369] T 41727d7c0bd94286aa763f8dc974e51d P 4742e9156ba346e4b9fcf79305c4eb44: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 05:17:09.458853 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:09.445021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56956 (local address 127.0.97.1:44299)
0928 05:17:09.445163 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:09.445168 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:09.446110 (+ 942us) server_negotiation.cc:407] Connection header received
0928 05:17:09.446314 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:09.446319 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:09.446398 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:09.446477 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:09.447811 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:09.448619 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:09.449842 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:09.450246 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:09.450350 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:09.454029 (+ 3679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:09.454041 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:09.454057 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:09.454089 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:09.456420 (+ 2331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:09.457016 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:09.457023 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:09.457026 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:09.457139 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:09.457481 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:09.457484 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:09.457486 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:09.457812 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:09.458080 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:09.458342 (+ 262us) server_negotiation.cc:299] Negotiation successful
0928 05:17:09.458531 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
Sep 28 05:17:09 dist-test-slave-whsn krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572629, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 05:17:09 dist-test-slave-whsn krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572629, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260928 05:17:09.497236 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:09.485560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56958 (local address 127.0.97.1:44299)
0928 05:17:09.485786 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:09.485792 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:09.485950 (+ 158us) server_negotiation.cc:407] Connection header received
0928 05:17:09.486094 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:09.486099 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:09.486156 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:09.486234 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:09.487338 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:09.488016 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:09.489261 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:09.489651 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:09.489739 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:09.493123 (+ 3384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:09.493128 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:09.493131 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:09.493159 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:09.495293 (+ 2134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:09.495790 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:09.495794 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:09.495796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:09.495905 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:09.496182 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:09.496185 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:09.496187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:09.496404 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:09.496580 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:09.496748 (+ 168us) server_negotiation.cc:299] Negotiation successful
0928 05:17:09.496941 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
W20260928 05:17:09.506556 1286 tablet.cc:2369] T 41727d7c0bd94286aa763f8dc974e51d P 77d62e02a38745cca1634e7f35ddf52c: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 05:17:09 dist-test-slave-whsn krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572629, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 05:17:09 dist-test-slave-whsn krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572629, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260928 05:17:09.542882 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:09.525847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56972 (local address 127.0.97.1:44299)
0928 05:17:09.526053 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:09.526060 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:09.526316 (+ 256us) server_negotiation.cc:407] Connection header received
0928 05:17:09.526529 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:09.526535 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:09.526626 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:09.526728 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:09.528324 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:09.529258 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:09.530740 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:09.531379 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:09.531519 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:09.535765 (+ 4246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:09.535772 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:09.535777 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:09.535818 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:09.540262 (+ 4444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:09.540936 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:09.540942 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:09.540946 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:09.541071 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:09.541398 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:09.541404 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:09.541408 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:09.541781 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:09.542037 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:09.542274 (+ 237us) server_negotiation.cc:299] Negotiation successful
0928 05:17:09.542506 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260928 05:17:09.550493 1066 tablet_service.cc:2908] Rejecting scan request for tablet 41727d7c0bd94286aa763f8dc974e51d: Uninitialized: clean time has not yet been initialized
I20260928 05:17:09.554071 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 1023
W20260928 05:17:09.557360 1152 tablet.cc:2369] T 41727d7c0bd94286aa763f8dc974e51d P 96917de6c0944f6bb1062b98fea22b40: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 05:17:09.565699 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:09.555207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56988 (local address 127.0.97.1:44299)
0928 05:17:09.555374 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:09.555380 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:09.555445 (+ 65us) server_negotiation.cc:407] Connection header received
0928 05:17:09.555624 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:09.555629 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:09.555699 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:09.555775 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:09.556807 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:09.557547 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:09.558729 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:09.559111 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:09.559220 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:09.561155 (+ 1935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:09.561161 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:09.561166 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:09.561194 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:09.563381 (+ 2187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:09.563954 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:09.563960 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:09.563963 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:09.564072 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:09.564383 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:09.564387 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:09.564391 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:09.564670 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:09.564949 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:09.565210 (+ 261us) server_negotiation.cc:299] Negotiation successful
0928 05:17:09.565404 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 05:17:09.566253 1146 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:17:09.651678 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 1023
I20260928 05:17:09.664577 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 1157
Sep 28 05:17:09 dist-test-slave-whsn krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572629, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260928 05:17:09.677809 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:09.665299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48772 (local address 127.0.97.2:38353)
0928 05:17:09.665456 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:09.665461 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:09.665478 (+ 17us) server_negotiation.cc:407] Connection header received
0928 05:17:09.665603 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:09.665607 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:09.665671 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:09.665749 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:09.666770 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:09.667451 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:09.668623 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:09.669113 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:09.669197 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:09.672667 (+ 3470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:09.672679 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:09.672694 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:09.672735 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:09.675361 (+ 2626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:09.675958 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:09.675965 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:09.675967 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:09.676101 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:09.676442 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:09.676447 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:09.676450 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:09.676767 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:09.677060 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:09.677311 (+ 251us) server_negotiation.cc:299] Negotiation successful
0928 05:17:09.677535 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 05:17:09.678422 1280 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:17:09.763262 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 1157
I20260928 05:17:09.776196 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 1290
Sep 28 05:17:09 dist-test-slave-whsn krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572629, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260928 05:17:09.793224 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:09.776840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44388 (local address 127.0.97.3:45693)
0928 05:17:09.776995 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:09.776999 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:09.777056 (+ 57us) server_negotiation.cc:407] Connection header received
0928 05:17:09.777195 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:09.777198 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:09.777268 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:09.777346 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:09.778456 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:09.779213 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:09.780411 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:09.780815 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:09.780975 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:09.788039 (+ 7064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:09.788055 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:09.788071 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:09.788107 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:09.790912 (+ 2805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:09.791520 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:09.791528 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:09.791530 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:09.791662 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:09.791969 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:09.791972 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:09.791974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:09.792307 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:09.792574 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:09.792780 (+ 206us) server_negotiation.cc:299] Negotiation successful
0928 05:17:09.792978 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 05:17:09.793819 1413 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:17:09.878089 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 1290
I20260928 05:17:09.890728 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 930
I20260928 05:17:09.891820 992 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:17:09.993853 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 930
2026-09-28T05:17:10Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (3745 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:17:10 dist-test-slave-whsn krb5kdc[1466](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:17:10 dist-test-slave-whsn krb5kdc[1466](info): set up 2 sockets
Sep 28 05:17:10 dist-test-slave-whsn krb5kdc[1466](info): commencing operation
krb5kdc: starting...
W20260928 05:17:10.972529 388 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.003s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/krb5kdc/test-user.keytab.
Sep 28 05:17:11 dist-test-slave-whsn krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:17:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:17:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 05:17:11.128315 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39279
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39279
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:17:11.423642 1479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:17:11.423915 1479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:17:11.424015 1479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:17:11.430986 1479 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:17:11.431087 1479 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 05:17:11.431129 1479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:17:11.431193 1479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:17:11.431247 1479 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:17:11.431310 1479 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:17:11.440596 1479 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39279
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:17:11.441766 1479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:17:11.443039 1479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:17:11.454352 1485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:11.454432 1487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:11.454427 1484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:17:11.455338 1479 server_base.cc:1034] running on GCE node
I20260928 05:17:11.456032 1479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:17:11.457315 1479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:17:11.458525 1479 hybrid_clock.cc:648] HybridClock initialized: now 1790572631458471 us; error 27 us; skew 500 ppm
I20260928 05:17:11.458834 1479 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:17:11 dist-test-slave-whsn krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572631, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:17:11.461658 1479 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 05:17:11.462941 1479 webserver.cc:460] Webserver started at http://127.0.97.62:33787/ using document root <none> and password file <none>
I20260928 05:17:11.463557 1479 fs_manager.cc:359] Metadata directory not provided
I20260928 05:17:11.463640 1479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:17:11.463927 1479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:17:11.466619 1479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/minicluster-data/master-0/data/instance:
uuid: "d09101969d894158a93c73549cbd740f"
format_stamp: "Formatted at 2026-09-28 05:17:11 on dist-test-slave-whsn"
server_key: "c074af51631dd1842022fde94e981f4f"
server_key_iv: "34b1f3c44ca89d869422fa86091ff105"
server_key_version: "encryptionkey@0"
I20260928 05:17:11.467321 1479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/minicluster-data/master-0/wal/instance:
uuid: "d09101969d894158a93c73549cbd740f"
format_stamp: "Formatted at 2026-09-28 05:17:11 on dist-test-slave-whsn"
server_key: "c074af51631dd1842022fde94e981f4f"
server_key_iv: "34b1f3c44ca89d869422fa86091ff105"
server_key_version: "encryptionkey@0"
I20260928 05:17:11.471860 1479 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260928 05:17:11.475255 1494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:11.476454 1479 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 05:17:11.476595 1479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/minicluster-data/master-0/wal
uuid: "d09101969d894158a93c73549cbd740f"
format_stamp: "Formatted at 2026-09-28 05:17:11 on dist-test-slave-whsn"
server_key: "c074af51631dd1842022fde94e981f4f"
server_key_iv: "34b1f3c44ca89d869422fa86091ff105"
server_key_version: "encryptionkey@0"
I20260928 05:17:11.476765 1479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:17:11.494211 1479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:17:11.497957 1479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:17:11.498170 1479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:17:11.515434 1479 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39279
I20260928 05:17:11.515456 1555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39279 every 8 connection(s)
I20260928 05:17:11.516582 1479 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:17:11.519944 1556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:17:11.521283 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 1479
I20260928 05:17:11.521486 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/minicluster-data/master-0/wal/instance
I20260928 05:17:11.522249 388 external_mini_cluster.cc:1428] Setting key ea5e857b4937fbae0a08d7c364b23565
I20260928 05:17:11.533196 1556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d09101969d894158a93c73549cbd740f: Bootstrap starting.
Sep 28 05:17:11 dist-test-slave-whsn krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:17:11.536752 1556 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d09101969d894158a93c73549cbd740f: Neither blocks nor log segments found. Creating new log.
I20260928 05:17:11.537840 1556 log.cc:824] T 00000000000000000000000000000000 P d09101969d894158a93c73549cbd740f: Log is configured to *not* fsync() on all Append() calls
I20260928 05:17:11.540578 1556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d09101969d894158a93c73549cbd740f: No bootstrap required, opened a new log
I20260928 05:17:11.543521 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:11.524160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46994 (local address 127.0.97.62:39279)
0928 05:17:11.524959 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:11.524972 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:11.525017 (+ 45us) server_negotiation.cc:407] Connection header received
0928 05:17:11.526508 (+ 1491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:11.526532 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:11.526888 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:11.527180 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:11.528740 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:11.530022 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:11.531686 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:11.532301 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:11.532415 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:11.536798 (+ 4383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:11.536805 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:11.536818 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:11.536890 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:11.539814 (+ 2924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:11.540550 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:11.540556 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:11.540561 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:11.540727 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:11.541118 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:11.541121 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:11.541123 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:11.541594 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:11.541784 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:11.542216 (+ 432us) server_negotiation.cc:299] Negotiation successful
0928 05:17:11.542550 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":370,"threads_started":1}
I20260928 05:17:11.548415 1556 raft_consensus.cc:348] T 00000000000000000000000000000000 P d09101969d894158a93c73549cbd740f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d09101969d894158a93c73549cbd740f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39279 } }
I20260928 05:17:11.548717 1556 raft_consensus.cc:374] T 00000000000000000000000000000000 P d09101969d894158a93c73549cbd740f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:17:11.548796 1556 raft_consensus.cc:729] T 00000000000000000000000000000000 P d09101969d894158a93c73549cbd740f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09101969d894158a93c73549cbd740f, State: Initialized, Role: FOLLOWER
I20260928 05:17:11.549382 1556 consensus_queue.cc:260] T 00000000000000000000000000000000 P d09101969d894158a93c73549cbd740f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d09101969d894158a93c73549cbd740f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39279 } }
I20260928 05:17:11.549618 1556 raft_consensus.cc:388] T 00000000000000000000000000000000 P d09101969d894158a93c73549cbd740f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:17:11.549711 1556 raft_consensus.cc:482] T 00000000000000000000000000000000 P d09101969d894158a93c73549cbd740f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:17:11.549823 1556 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d09101969d894158a93c73549cbd740f [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:17:11.551695 1556 raft_consensus.cc:504] T 00000000000000000000000000000000 P d09101969d894158a93c73549cbd740f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d09101969d894158a93c73549cbd740f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39279 } }
I20260928 05:17:11.552099 1556 leader_election.cc:302] T 00000000000000000000000000000000 P d09101969d894158a93c73549cbd740f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d09101969d894158a93c73549cbd740f; no voters:
I20260928 05:17:11.552634 1556 leader_election.cc:288] T 00000000000000000000000000000000 P d09101969d894158a93c73549cbd740f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:17:11.552747 1561 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d09101969d894158a93c73549cbd740f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:17:11.553480 1561 raft_consensus.cc:686] T 00000000000000000000000000000000 P d09101969d894158a93c73549cbd740f [term 1 LEADER]: Becoming Leader. State: Replica: d09101969d894158a93c73549cbd740f, State: Running, Role: LEADER
I20260928 05:17:11.554029 1561 consensus_queue.cc:237] T 00000000000000000000000000000000 P d09101969d894158a93c73549cbd740f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d09101969d894158a93c73549cbd740f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39279 } }
I20260928 05:17:11.554589 1556 sys_catalog.cc:564] T 00000000000000000000000000000000 P d09101969d894158a93c73549cbd740f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:17:11.556615 1562 sys_catalog.cc:455] T 00000000000000000000000000000000 P d09101969d894158a93c73549cbd740f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d09101969d894158a93c73549cbd740f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d09101969d894158a93c73549cbd740f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39279 } } }
I20260928 05:17:11.557125 1562 sys_catalog.cc:458] T 00000000000000000000000000000000 P d09101969d894158a93c73549cbd740f [sys.catalog]: This master's current role is: LEADER
I20260928 05:17:11.557117 1563 sys_catalog.cc:455] T 00000000000000000000000000000000 P d09101969d894158a93c73549cbd740f [sys.catalog]: SysCatalogTable state changed. Reason: New leader d09101969d894158a93c73549cbd740f. Latest consensus state: current_term: 1 leader_uuid: "d09101969d894158a93c73549cbd740f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d09101969d894158a93c73549cbd740f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39279 } } }
I20260928 05:17:11.558089 1563 sys_catalog.cc:458] T 00000000000000000000000000000000 P d09101969d894158a93c73549cbd740f [sys.catalog]: This master's current role is: LEADER
I20260928 05:17:11.558909 1570 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:17:11.563511 1570 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:17:11.571414 1570 catalog_manager.cc:1347] Generated new cluster ID: ea369592d7ec41dbafb52488e3b383b5
I20260928 05:17:11.571513 1570 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:17:11.606258 1570 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:17:11.608127 1570 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:17:11.620242 1570 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d09101969d894158a93c73549cbd740f: Generated new TSK 0
I20260928 05:17:11.621289 1570 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260928 05:17:11.665071 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39279
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:17:11.966511 1582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:17:11.966789 1582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:17:11.966919 1582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:17:11.973994 1582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:17:11.974100 1582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:17:11.974216 1582 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 05:17:11.974318 1582 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 05:17:11.983932 1582 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:39279
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:17:11.985169 1582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:17:11.986399 1582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:17:11.998306 1590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:11.998301 1588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:11.998399 1587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:17:11.999105 1582 server_base.cc:1034] running on GCE node
I20260928 05:17:11.999650 1582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:17:12.000474 1582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:17:12.001680 1582 hybrid_clock.cc:648] HybridClock initialized: now 1790572632001620 us; error 37 us; skew 500 ppm
I20260928 05:17:12.001940 1582 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:17:11 dist-test-slave-whsn krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572631, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:17:12.004909 1582 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260928 05:17:12.006229 1582 webserver.cc:460] Webserver started at http://127.0.97.1:41631/ using document root <none> and password file <none>
I20260928 05:17:12.006870 1582 fs_manager.cc:359] Metadata directory not provided
I20260928 05:17:12.006963 1582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:17:12.007253 1582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:17:12.009936 1582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/minicluster-data/ts-0/data/instance:
uuid: "9c6a93ea324e4afc960e099ff730e232"
format_stamp: "Formatted at 2026-09-28 05:17:12 on dist-test-slave-whsn"
server_key: "8ef0130965d5334db20e2e942856551e"
server_key_iv: "a415101e516c1e87f76b3a239d07d259"
server_key_version: "encryptionkey@0"
I20260928 05:17:12.010655 1582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/minicluster-data/ts-0/wal/instance:
uuid: "9c6a93ea324e4afc960e099ff730e232"
format_stamp: "Formatted at 2026-09-28 05:17:12 on dist-test-slave-whsn"
server_key: "8ef0130965d5334db20e2e942856551e"
server_key_iv: "a415101e516c1e87f76b3a239d07d259"
server_key_version: "encryptionkey@0"
I20260928 05:17:12.015264 1582 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 05:17:12.018402 1597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:12.019574 1582 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 05:17:12.019717 1582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/minicluster-data/ts-0/wal
uuid: "9c6a93ea324e4afc960e099ff730e232"
format_stamp: "Formatted at 2026-09-28 05:17:12 on dist-test-slave-whsn"
server_key: "8ef0130965d5334db20e2e942856551e"
server_key_iv: "a415101e516c1e87f76b3a239d07d259"
server_key_version: "encryptionkey@0"
I20260928 05:17:12.019897 1582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:17:12.046240 1582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:17:12.050287 1582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:17:12.050544 1582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:17:12.052968 1582 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:17:12.053064 1582 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:12.053190 1582 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:17:12.053252 1582 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 05:17:12 dist-test-slave-whsn krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572631, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:17:12.073352 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:12.053657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38439 (local address 127.0.97.62:39279)
0928 05:17:12.054143 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:12.054149 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:12.055391 (+ 1242us) server_negotiation.cc:407] Connection header received
0928 05:17:12.057488 (+ 2097us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:12.057493 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:12.057573 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:12.057815 (+ 242us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:12.059919 (+ 2104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:12.060838 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:12.063147 (+ 2309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:12.063674 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:12.063785 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:12.067617 (+ 3832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:12.067622 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:12.067625 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:12.067655 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:12.070454 (+ 2799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:12.071271 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:12.071276 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:12.071279 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:12.071391 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:12.071769 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:12.071774 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:12.071778 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:12.072032 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:12.072164 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:12.072775 (+ 611us) server_negotiation.cc:299] Negotiation successful
0928 05:17:12.073014 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":306,"threads_started":1}
I20260928 05:17:12.074278 1607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:12.054396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39279 (local address 127.0.97.1:38439)
0928 05:17:12.055239 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:12.055291 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:12.057228 (+ 1937us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:12.058212 (+ 984us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:12.058221 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:12.058924 (+ 703us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:17:12.059752 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:12.059777 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:12.061002 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:12.061008 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:12.063007 (+ 1999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:12.063019 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:12.063907 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:12.063915 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:12.064152 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:12.064912 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:17:12.064942 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:17:12.067417 (+ 2475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:17:12.070562 (+ 3145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:12.070571 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:12.070589 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:12.071170 (+ 581us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:12.071492 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:12.071497 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:12.071501 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:12.071673 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:12.072149 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:17:12.072156 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:17:12.072455 (+ 299us) client_negotiation.cc:769] Sending connection context
0928 05:17:12.072683 (+ 228us) client_negotiation.cc:240] Negotiation successful
0928 05:17:12.073189 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":364,"threads_started":1}
I20260928 05:17:12.094736 1663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:12.081123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39279 (local address 127.0.97.1:36545)
0928 05:17:12.081831 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:12.081849 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:12.081943 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:12.082270 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:12.082273 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:12.082648 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:17:12.082898 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:12.082905 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:12.084385 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:12.084389 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:12.085601 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:12.085612 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:12.086426 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:12.086432 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:12.086596 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:12.087206 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:17:12.087235 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:17:12.088377 (+ 1142us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:17:12.091552 (+ 3175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:12.091557 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:12.091561 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:12.091996 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:12.092485 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:12.092490 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:12.092493 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:12.092668 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:12.093419 (+ 751us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:17:12.093426 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:17:12.093657 (+ 231us) client_negotiation.cc:769] Sending connection context
0928 05:17:12.093801 (+ 144us) client_negotiation.cc:240] Negotiation successful
0928 05:17:12.094172 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":432,"threads_started":1}
I20260928 05:17:12.095887 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:12.081038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36545 (local address 127.0.97.62:39279)
0928 05:17:12.081195 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:12.081200 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:12.081932 (+ 732us) server_negotiation.cc:407] Connection header received
0928 05:17:12.082050 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:12.082057 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:12.082143 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:12.082272 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:12.083030 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:12.084234 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:12.085718 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:12.086263 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:12.086368 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:12.088560 (+ 2192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:12.088564 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:12.088568 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:12.088596 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:12.091368 (+ 2772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:12.092125 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:12.092132 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:12.092137 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:12.092319 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:12.092800 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:12.092806 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:12.092811 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:12.093251 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:12.095068 (+ 1817us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:12.095212 (+ 144us) server_negotiation.cc:299] Negotiation successful
0928 05:17:12.095463 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 05:17:12.107861 1582 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37715
I20260928 05:17:12.107868 1717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37715 every 8 connection(s)
I20260928 05:17:12.109076 1582 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/minicluster-data/ts-0/data/info.pb
I20260928 05:17:12.110932 1718 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39279
I20260928 05:17:12.111126 1718 heartbeater.cc:464] Registering TS with master...
I20260928 05:17:12.111665 1718 heartbeater.cc:511] Master 127.0.97.62:39279 requested a full tablet report, sending...
I20260928 05:17:12.113541 1511 ts_manager.cc:194] Registered new tserver with Master: 9c6a93ea324e4afc960e099ff730e232 (127.0.97.1:37715)
I20260928 05:17:12.114995 1511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:38439
I20260928 05:17:12.118799 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 1582
I20260928 05:17:12.118955 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/minicluster-data/ts-0/wal/instance
I20260928 05:17:12.119486 388 external_mini_cluster.cc:1428] Setting key a4da39234fff1967982404be027c7f34
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/krb5kdc/kudu_127.0.97.2.keytab.
W20260928 05:17:12.243335 388 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.97.2: real 0.122s user 0.002s sys 0.005s
I20260928 05:17:12.244002 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39279
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:17:12.561178 1722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:17:12.561450 1722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:17:12.561583 1722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:17:12.568681 1722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:17:12.568804 1722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:17:12.568944 1722 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 05:17:12.569056 1722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 05:17:12.578593 1722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:39279
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:17:12.579780 1722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:17:12.581080 1722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:17:12.593101 1727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:12.593169 1730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:12.593258 1728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:17:12.594066 1722 server_base.cc:1034] running on GCE node
I20260928 05:17:12.594568 1722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:17:12.595422 1722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:17:12.596633 1722 hybrid_clock.cc:648] HybridClock initialized: now 1790572632596568 us; error 38 us; skew 500 ppm
I20260928 05:17:12.596926 1722 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:17:12 dist-test-slave-whsn krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572632, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:17:12.599694 1722 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260928 05:17:12.601060 1722 webserver.cc:460] Webserver started at http://127.0.97.2:42077/ using document root <none> and password file <none>
I20260928 05:17:12.601696 1722 fs_manager.cc:359] Metadata directory not provided
I20260928 05:17:12.601785 1722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:17:12.602066 1722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:17:12.604811 1722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/minicluster-data/ts-1/data/instance:
uuid: "c72c1d866ac34aa3bbcae200c28f8eb3"
format_stamp: "Formatted at 2026-09-28 05:17:12 on dist-test-slave-whsn"
server_key: "68dde5018dbe98a6eec5a9a45dc28acd"
server_key_iv: "a097ed07458249b6be2c65a41b0c965f"
server_key_version: "encryptionkey@0"
I20260928 05:17:12.605644 1722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/minicluster-data/ts-1/wal/instance:
uuid: "c72c1d866ac34aa3bbcae200c28f8eb3"
format_stamp: "Formatted at 2026-09-28 05:17:12 on dist-test-slave-whsn"
server_key: "68dde5018dbe98a6eec5a9a45dc28acd"
server_key_iv: "a097ed07458249b6be2c65a41b0c965f"
server_key_version: "encryptionkey@0"
I20260928 05:17:12.610494 1722 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260928 05:17:12.613686 1737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:12.614921 1722 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 05:17:12.615063 1722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/minicluster-data/ts-1/wal
uuid: "c72c1d866ac34aa3bbcae200c28f8eb3"
format_stamp: "Formatted at 2026-09-28 05:17:12 on dist-test-slave-whsn"
server_key: "68dde5018dbe98a6eec5a9a45dc28acd"
server_key_iv: "a097ed07458249b6be2c65a41b0c965f"
server_key_version: "encryptionkey@0"
I20260928 05:17:12.615255 1722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:17:12.631793 1722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:17:12.635434 1722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:17:12.635665 1722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:17:12.638160 1722 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:17:12.638255 1722 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:12.638386 1722 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:17:12.638463 1722 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 05:17:12 dist-test-slave-whsn krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572632, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:17:12.660738 1747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:12.639238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39279 (local address 127.0.97.2:43651)
0928 05:17:12.640013 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:12.640047 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:12.641822 (+ 1775us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:12.642578 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:12.642591 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:12.643438 (+ 847us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:17:12.644379 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:12.644405 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:12.645703 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:12.645710 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:12.648104 (+ 2394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:12.648124 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:12.649014 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:12.649020 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:12.649268 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:12.650019 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:17:12.650052 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:17:12.652997 (+ 2945us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:17:12.656573 (+ 3576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:12.656584 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:12.656601 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:12.657243 (+ 642us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:12.657619 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:12.657624 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:12.657628 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:12.657902 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:12.658603 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:17:12.658609 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:17:12.658920 (+ 311us) client_negotiation.cc:769] Sending connection context
0928 05:17:12.659260 (+ 340us) client_negotiation.cc:240] Negotiation successful
0928 05:17:12.659667 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":356,"threads_started":1}
I20260928 05:17:12.661592 1746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:12.638594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43651 (local address 127.0.97.62:39279)
0928 05:17:12.639266 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:12.639271 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:12.640155 (+ 884us) server_negotiation.cc:407] Connection header received
0928 05:17:12.642091 (+ 1936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:12.642097 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:12.642175 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:12.642289 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:12.644576 (+ 2287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:12.645520 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:12.648264 (+ 2744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:12.648771 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:12.649498 (+ 727us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:12.653204 (+ 3706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:12.653209 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:12.653212 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:12.653243 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:12.656438 (+ 3195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:12.657371 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:12.657377 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:12.657380 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:12.657520 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:12.658072 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:12.658078 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:12.658081 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:12.658480 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:12.658638 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:12.661183 (+ 2545us) server_negotiation.cc:299] Negotiation successful
0928 05:17:12.661342 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":489,"threads_started":1}
I20260928 05:17:12.679105 1746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:12.666604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36555 (local address 127.0.97.62:39279)
0928 05:17:12.666745 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:12.666749 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:12.667227 (+ 478us) server_negotiation.cc:407] Connection header received
0928 05:17:12.667376 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:12.667381 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:12.667443 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:12.667538 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:12.668935 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:12.669880 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:12.671112 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:12.671648 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:12.671751 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:12.673959 (+ 2208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:12.673965 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:12.673970 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:12.673999 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:12.676662 (+ 2663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:12.677503 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:12.677508 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:12.677512 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:12.677623 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:12.677962 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:12.677965 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:12.677968 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:12.678232 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:12.678333 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:12.678745 (+ 412us) server_negotiation.cc:299] Negotiation successful
0928 05:17:12.678886 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 05:17:12.679452 1820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:12.666675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39279 (local address 127.0.97.2:36555)
0928 05:17:12.667142 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:12.667156 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:12.667274 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:12.667689 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:12.667696 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:12.668293 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:17:12.668742 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:12.668758 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:12.670003 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:12.670009 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:12.670991 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:12.671002 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:12.671814 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:12.671821 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:12.671986 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:12.672554 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:17:12.672581 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:17:12.673777 (+ 1196us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:17:12.676776 (+ 2999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:12.676782 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:12.676786 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:12.677379 (+ 593us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:12.677717 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:12.677723 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:12.677727 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:12.677875 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:12.678323 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:17:12.678328 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:17:12.678520 (+ 192us) client_negotiation.cc:769] Sending connection context
0928 05:17:12.678654 (+ 134us) client_negotiation.cc:240] Negotiation successful
0928 05:17:12.678964 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":249,"threads_started":1}
I20260928 05:17:12.683357 1722 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36695
I20260928 05:17:12.683359 1857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36695 every 8 connection(s)
I20260928 05:17:12.684463 1722 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/minicluster-data/ts-1/data/info.pb
I20260928 05:17:12.686239 1858 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39279
I20260928 05:17:12.686416 1858 heartbeater.cc:464] Registering TS with master...
I20260928 05:17:12.686952 1858 heartbeater.cc:511] Master 127.0.97.62:39279 requested a full tablet report, sending...
I20260928 05:17:12.688051 1511 ts_manager.cc:194] Registered new tserver with Master: c72c1d866ac34aa3bbcae200c28f8eb3 (127.0.97.2:36695)
I20260928 05:17:12.688771 1511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:43651
I20260928 05:17:12.688921 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 1722
I20260928 05:17:12.689079 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/minicluster-data/ts-1/wal/instance
I20260928 05:17:12.689512 388 external_mini_cluster.cc:1428] Setting key 42f7cf2ba794b28cc4ef838e77e8a0e7
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260928 05:17:12.723421 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39279
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:17:13.020182 1862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:17:13.020450 1862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:17:13.020583 1862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:17:13.027644 1862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:17:13.027755 1862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:17:13.027874 1862 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 05:17:13.027988 1862 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 05:17:13.037288 1862 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:39279
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:17:13.038478 1862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:17:13.039726 1862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:17:13.052114 1867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:13.052052 1870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:13.052032 1868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:17:13.053053 1862 server_base.cc:1034] running on GCE node
I20260928 05:17:13.053619 1862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:17:13.054543 1862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:17:13.055761 1862 hybrid_clock.cc:648] HybridClock initialized: now 1790572633055675 us; error 54 us; skew 500 ppm
I20260928 05:17:13.056071 1862 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:17:13 dist-test-slave-whsn krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572633, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:17:13.058941 1862 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260928 05:17:13.060286 1862 webserver.cc:460] Webserver started at http://127.0.97.3:33133/ using document root <none> and password file <none>
I20260928 05:17:13.060976 1862 fs_manager.cc:359] Metadata directory not provided
I20260928 05:17:13.061070 1862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:17:13.061359 1862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:17:13.064185 1862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/minicluster-data/ts-2/data/instance:
uuid: "757e3e76bff144399dc79b6855581575"
format_stamp: "Formatted at 2026-09-28 05:17:13 on dist-test-slave-whsn"
server_key: "396fc3453449709646e25c6f57c32219"
server_key_iv: "c75c534189a88cde64406994515083d4"
server_key_version: "encryptionkey@0"
I20260928 05:17:13.064978 1862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/minicluster-data/ts-2/wal/instance:
uuid: "757e3e76bff144399dc79b6855581575"
format_stamp: "Formatted at 2026-09-28 05:17:13 on dist-test-slave-whsn"
server_key: "396fc3453449709646e25c6f57c32219"
server_key_iv: "c75c534189a88cde64406994515083d4"
server_key_version: "encryptionkey@0"
I20260928 05:17:13.069563 1862 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 05:17:13.072613 1877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:13.073800 1862 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260928 05:17:13.073935 1862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/minicluster-data/ts-2/wal
uuid: "757e3e76bff144399dc79b6855581575"
format_stamp: "Formatted at 2026-09-28 05:17:13 on dist-test-slave-whsn"
server_key: "396fc3453449709646e25c6f57c32219"
server_key_iv: "c75c534189a88cde64406994515083d4"
server_key_version: "encryptionkey@0"
I20260928 05:17:13.074132 1862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:17:13.089598 1862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:17:13.093254 1862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:17:13.093489 1862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:17:13.096360 1862 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:17:13.096455 1862 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:13.096576 1862 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:17:13.096634 1862 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 05:17:13 dist-test-slave-whsn krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572633, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:17:13.116528 1746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:13.097175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52417 (local address 127.0.97.62:39279)
0928 05:17:13.097341 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:13.097348 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:13.099074 (+ 1726us) server_negotiation.cc:407] Connection header received
0928 05:17:13.101416 (+ 2342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:13.101423 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:13.101512 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:13.101620 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:13.103035 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:13.104178 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:13.106425 (+ 2247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:13.107051 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:13.107194 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:13.110680 (+ 3486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:13.110685 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:13.110688 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:13.110720 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:13.113461 (+ 2741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:13.114144 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:13.114149 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:13.114151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:13.114277 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:13.114705 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:13.114711 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:13.114715 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:13.115079 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:13.115217 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:13.115908 (+ 691us) server_negotiation.cc:299] Negotiation successful
0928 05:17:13.116132 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 05:17:13.117489 1718 heartbeater.cc:503] Master 127.0.97.62:39279 was elected leader, sending a full tablet report...
I20260928 05:17:13.117619 1886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:13.097790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39279 (local address 127.0.97.3:52417)
0928 05:17:13.098857 (+ 1067us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:13.098942 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:13.101079 (+ 2137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:13.101824 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:13.101832 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:13.102319 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:17:13.102880 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:13.102897 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:13.104308 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:13.104312 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:13.106180 (+ 1868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:13.106196 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:13.107291 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:13.107299 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:13.107527 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:13.108173 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:17:13.108197 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:17:13.110458 (+ 2261us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:17:13.113593 (+ 3135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:13.113602 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:13.113617 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:13.114018 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:13.114381 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:13.114387 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:13.114391 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:13.114579 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:13.115214 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:17:13.115223 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:17:13.115549 (+ 326us) client_negotiation.cc:769] Sending connection context
0928 05:17:13.115773 (+ 224us) client_negotiation.cc:240] Negotiation successful
0928 05:17:13.116231 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":762,"thread_start_us":446,"threads_started":1}
I20260928 05:17:13.136078 1746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:13.122810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54015 (local address 127.0.97.62:39279)
0928 05:17:13.122964 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:13.122968 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:13.123543 (+ 575us) server_negotiation.cc:407] Connection header received
0928 05:17:13.123748 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:13.123753 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:13.123832 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:13.123905 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:13.125061 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:13.125977 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:13.127808 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:13.128344 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:13.128476 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:13.130666 (+ 2190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:13.130672 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:13.130677 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:13.130711 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:13.133572 (+ 2861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:13.134357 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:13.134362 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:13.134366 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:13.134494 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:13.134807 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:13.134812 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:13.134816 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:13.135072 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:13.135193 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:13.135564 (+ 371us) server_negotiation.cc:299] Negotiation successful
0928 05:17:13.135746 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 05:17:13.136178 1911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:13.122873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39279 (local address 127.0.97.3:54015)
0928 05:17:13.123446 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:13.123461 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:13.123635 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:13.123954 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:13.123959 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:13.124529 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:17:13.124873 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:13.124898 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:13.126183 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:13.126188 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:13.127680 (+ 1492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:13.127697 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:13.128534 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:13.128545 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:13.128734 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:13.129462 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:17:13.129483 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:17:13.130451 (+ 968us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:17:13.133716 (+ 3265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:13.133720 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:13.133722 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:13.134255 (+ 533us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:13.134589 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:13.134594 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:13.134598 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:13.134715 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:13.135175 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:17:13.135180 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:17:13.135343 (+ 163us) client_negotiation.cc:769] Sending connection context
0928 05:17:13.135457 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 05:17:13.135743 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":342,"threads_started":1}
I20260928 05:17:13.149935 1862 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43191
I20260928 05:17:13.149963 1996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43191 every 8 connection(s)
I20260928 05:17:13.151046 1862 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/minicluster-data/ts-2/data/info.pb
I20260928 05:17:13.152943 1997 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39279
I20260928 05:17:13.153139 1997 heartbeater.cc:464] Registering TS with master...
I20260928 05:17:13.153724 1997 heartbeater.cc:511] Master 127.0.97.62:39279 requested a full tablet report, sending...
I20260928 05:17:13.154954 1510 ts_manager.cc:194] Registered new tserver with Master: 757e3e76bff144399dc79b6855581575 (127.0.97.3:43191)
I20260928 05:17:13.155812 1510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:52417
I20260928 05:17:13.156342 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 1862
I20260928 05:17:13.156486 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/minicluster-data/ts-2/wal/instance
I20260928 05:17:13.156920 388 external_mini_cluster.cc:1428] Setting key 1345e96f1e635abc6cc876457de90833
I20260928 05:17:13.159879 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 05:17:13 dist-test-slave-whsn krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572633, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 05:17:13 dist-test-slave-whsn krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572633, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:17:13.198907 1746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:13.187015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55514 (local address 127.0.97.62:39279)
0928 05:17:13.187228 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:13.187234 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:13.187405 (+ 171us) server_negotiation.cc:407] Connection header received
0928 05:17:13.187547 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:13.187550 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:13.187611 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:13.187691 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:13.188942 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:13.189674 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:13.190873 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:13.191268 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:13.191364 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:13.194618 (+ 3254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:13.194623 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:13.194626 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:13.194656 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:13.196900 (+ 2244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:13.197437 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:13.197441 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:13.197443 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:13.197551 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:13.197829 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:13.197832 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:13.197834 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:13.198099 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:13.198272 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:13.198455 (+ 183us) server_negotiation.cc:299] Negotiation successful
0928 05:17:13.198604 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 05:17:13.202485 1510 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55514:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 05:17:13.209041 1510 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 05:17:13.241317 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:13.220204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54108 (local address 127.0.97.3:43191)
0928 05:17:13.232479 (+ 12275us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:13.232489 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:13.232525 (+ 36us) server_negotiation.cc:407] Connection header received
0928 05:17:13.232631 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:13.232640 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:13.232939 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:13.234114 (+ 1175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:17:13.234402 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:13.235903 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:13.239233 (+ 3330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:13.240151 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:13.240269 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:13.240519 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:13.240811 (+ 292us) server_negotiation.cc:299] Negotiation successful
0928 05:17:13.240989 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12086,"thread_start_us":12892,"threads_started":1}
I20260928 05:17:13.242286 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:13.219951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43191 (local address 127.0.0.1:54108)
0928 05:17:13.225372 (+ 5421us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:13.225416 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:13.225561 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:13.233385 (+ 7824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:13.233394 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:13.233429 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:17:13.233880 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:13.233892 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:13.236684 (+ 2792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:13.236690 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:13.239070 (+ 2380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:13.239087 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:13.240331 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:13.240337 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:13.240604 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:13.240641 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 05:17:13.240756 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 05:17:13.241891 (+ 1135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5203,"mutex_wait_us":43,"thread_start_us":15430,"threads_started":1}
I20260928 05:17:13.242852 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:13.220543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41062 (local address 127.0.97.1:37715)
0928 05:17:13.221075 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:13.221084 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:13.235004 (+ 13920us) server_negotiation.cc:407] Connection header received
0928 05:17:13.235805 (+ 801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:13.235813 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:13.236069 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:13.236229 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:17:13.237131 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:13.238253 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:13.240466 (+ 2213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:13.241885 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:13.242004 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:13.242189 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:13.242408 (+ 219us) server_negotiation.cc:299] Negotiation successful
0928 05:17:13.242583 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":232,"threads_started":1}
I20260928 05:17:13.243114 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:13.220064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37715 (local address 127.0.0.1:41062)
0928 05:17:13.234890 (+ 14826us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:13.234908 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:13.235625 (+ 717us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:13.236457 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:13.236461 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:13.236491 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:17:13.236938 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:13.236950 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:13.238399 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:13.238402 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:13.240276 (+ 1874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:13.240290 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:13.242070 (+ 1780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:13.242076 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:13.242231 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:13.242266 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 05:17:13.242534 (+ 268us) client_negotiation.cc:240] Negotiation successful
0928 05:17:13.242620 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14696,"mutex_wait_us":45,"spinlock_wait_cycles":1792,"thread_start_us":4959,"threads_started":1}
I20260928 05:17:13.243577 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:13.225930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53492 (local address 127.0.97.2:36695)
0928 05:17:13.235362 (+ 9432us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:13.235370 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:13.236464 (+ 1094us) server_negotiation.cc:407] Connection header received
0928 05:17:13.236624 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:13.236629 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:13.236840 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:13.237071 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:17:13.237692 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:13.239044 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:13.241434 (+ 2390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:13.242630 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:13.242753 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:13.242928 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:13.243186 (+ 258us) server_negotiation.cc:299] Negotiation successful
0928 05:17:13.243336 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9286,"thread_start_us":10050,"threads_started":1}
I20260928 05:17:13.243587 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:13.225898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36695 (local address 127.0.0.1:53492)
0928 05:17:13.236364 (+ 10466us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:13.236379 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:13.236498 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:13.237064 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:13.237068 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:13.237091 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:17:13.237530 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:13.237541 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:13.239180 (+ 1639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:13.239184 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:13.241257 (+ 2073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:13.241271 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:13.242821 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:13.242827 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:13.243003 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:13.243032 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 05:17:13.243126 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 05:17:13.243241 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10362,"mutex_wait_us":53,"spinlock_wait_cycles":11648,"thread_start_us":10240,"threads_started":1}
I20260928 05:17:13.246552 1648 tablet_service.cc:1467] Processing CreateTablet for tablet b9a7de9525da425e88427a6fe8793530 (DEFAULT_TABLE table=test-table [id=4d5336f4ba314fc5b180312e7da24187]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:17:13.246657 1932 tablet_service.cc:1467] Processing CreateTablet for tablet b9a7de9525da425e88427a6fe8793530 (DEFAULT_TABLE table=test-table [id=4d5336f4ba314fc5b180312e7da24187]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:17:13.247172 1788 tablet_service.cc:1467] Processing CreateTablet for tablet b9a7de9525da425e88427a6fe8793530 (DEFAULT_TABLE table=test-table [id=4d5336f4ba314fc5b180312e7da24187]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:17:13.247829 1648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9a7de9525da425e88427a6fe8793530. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:17:13.248026 1932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9a7de9525da425e88427a6fe8793530. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:17:13.248274 1788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9a7de9525da425e88427a6fe8793530. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:17:13.258548 2016 tablet_bootstrap.cc:492] T b9a7de9525da425e88427a6fe8793530 P 9c6a93ea324e4afc960e099ff730e232: Bootstrap starting.
I20260928 05:17:13.261365 2017 tablet_bootstrap.cc:492] T b9a7de9525da425e88427a6fe8793530 P c72c1d866ac34aa3bbcae200c28f8eb3: Bootstrap starting.
I20260928 05:17:13.262271 2016 tablet_bootstrap.cc:654] T b9a7de9525da425e88427a6fe8793530 P 9c6a93ea324e4afc960e099ff730e232: Neither blocks nor log segments found. Creating new log.
I20260928 05:17:13.262272 2018 tablet_bootstrap.cc:492] T b9a7de9525da425e88427a6fe8793530 P 757e3e76bff144399dc79b6855581575: Bootstrap starting.
I20260928 05:17:13.263669 2016 log.cc:824] T b9a7de9525da425e88427a6fe8793530 P 9c6a93ea324e4afc960e099ff730e232: Log is configured to *not* fsync() on all Append() calls
I20260928 05:17:13.264514 2017 tablet_bootstrap.cc:654] T b9a7de9525da425e88427a6fe8793530 P c72c1d866ac34aa3bbcae200c28f8eb3: Neither blocks nor log segments found. Creating new log.
I20260928 05:17:13.265430 2017 log.cc:824] T b9a7de9525da425e88427a6fe8793530 P c72c1d866ac34aa3bbcae200c28f8eb3: Log is configured to *not* fsync() on all Append() calls
I20260928 05:17:13.265836 2018 tablet_bootstrap.cc:654] T b9a7de9525da425e88427a6fe8793530 P 757e3e76bff144399dc79b6855581575: Neither blocks nor log segments found. Creating new log.
I20260928 05:17:13.266657 2016 tablet_bootstrap.cc:492] T b9a7de9525da425e88427a6fe8793530 P 9c6a93ea324e4afc960e099ff730e232: No bootstrap required, opened a new log
I20260928 05:17:13.266906 2016 ts_tablet_manager.cc:1397] T b9a7de9525da425e88427a6fe8793530 P 9c6a93ea324e4afc960e099ff730e232: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260928 05:17:13.266893 2018 log.cc:824] T b9a7de9525da425e88427a6fe8793530 P 757e3e76bff144399dc79b6855581575: Log is configured to *not* fsync() on all Append() calls
I20260928 05:17:13.267941 2017 tablet_bootstrap.cc:492] T b9a7de9525da425e88427a6fe8793530 P c72c1d866ac34aa3bbcae200c28f8eb3: No bootstrap required, opened a new log
I20260928 05:17:13.268195 2017 ts_tablet_manager.cc:1397] T b9a7de9525da425e88427a6fe8793530 P c72c1d866ac34aa3bbcae200c28f8eb3: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260928 05:17:13.270344 2018 tablet_bootstrap.cc:492] T b9a7de9525da425e88427a6fe8793530 P 757e3e76bff144399dc79b6855581575: No bootstrap required, opened a new log
I20260928 05:17:13.270584 2018 ts_tablet_manager.cc:1397] T b9a7de9525da425e88427a6fe8793530 P 757e3e76bff144399dc79b6855581575: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260928 05:17:13.275152 2017 raft_consensus.cc:348] T b9a7de9525da425e88427a6fe8793530 P c72c1d866ac34aa3bbcae200c28f8eb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "757e3e76bff144399dc79b6855581575" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43191 } } peers { permanent_uuid: "9c6a93ea324e4afc960e099ff730e232" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37715 } } peers { permanent_uuid: "c72c1d866ac34aa3bbcae200c28f8eb3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36695 } }
I20260928 05:17:13.275516 2017 raft_consensus.cc:374] T b9a7de9525da425e88427a6fe8793530 P c72c1d866ac34aa3bbcae200c28f8eb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:17:13.275611 2017 raft_consensus.cc:729] T b9a7de9525da425e88427a6fe8793530 P c72c1d866ac34aa3bbcae200c28f8eb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c72c1d866ac34aa3bbcae200c28f8eb3, State: Initialized, Role: FOLLOWER
I20260928 05:17:13.276156 2017 consensus_queue.cc:260] T b9a7de9525da425e88427a6fe8793530 P c72c1d866ac34aa3bbcae200c28f8eb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "757e3e76bff144399dc79b6855581575" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43191 } } peers { permanent_uuid: "9c6a93ea324e4afc960e099ff730e232" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37715 } } peers { permanent_uuid: "c72c1d866ac34aa3bbcae200c28f8eb3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36695 } }
I20260928 05:17:13.277192 1858 heartbeater.cc:503] Master 127.0.97.62:39279 was elected leader, sending a full tablet report...
I20260928 05:17:13.277324 2016 raft_consensus.cc:348] T b9a7de9525da425e88427a6fe8793530 P 9c6a93ea324e4afc960e099ff730e232 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "757e3e76bff144399dc79b6855581575" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43191 } } peers { permanent_uuid: "9c6a93ea324e4afc960e099ff730e232" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37715 } } peers { permanent_uuid: "c72c1d866ac34aa3bbcae200c28f8eb3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36695 } }
I20260928 05:17:13.277688 2016 raft_consensus.cc:374] T b9a7de9525da425e88427a6fe8793530 P 9c6a93ea324e4afc960e099ff730e232 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:17:13.277760 2016 raft_consensus.cc:729] T b9a7de9525da425e88427a6fe8793530 P 9c6a93ea324e4afc960e099ff730e232 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c6a93ea324e4afc960e099ff730e232, State: Initialized, Role: FOLLOWER
I20260928 05:17:13.277879 2017 ts_tablet_manager.cc:1428] T b9a7de9525da425e88427a6fe8793530 P c72c1d866ac34aa3bbcae200c28f8eb3: Time spent starting tablet: real 0.010s user 0.008s sys 0.003s
I20260928 05:17:13.278453 2016 consensus_queue.cc:260] T b9a7de9525da425e88427a6fe8793530 P 9c6a93ea324e4afc960e099ff730e232 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "757e3e76bff144399dc79b6855581575" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43191 } } peers { permanent_uuid: "9c6a93ea324e4afc960e099ff730e232" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37715 } } peers { permanent_uuid: "c72c1d866ac34aa3bbcae200c28f8eb3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36695 } }
I20260928 05:17:13.280313 2018 raft_consensus.cc:348] T b9a7de9525da425e88427a6fe8793530 P 757e3e76bff144399dc79b6855581575 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "757e3e76bff144399dc79b6855581575" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43191 } } peers { permanent_uuid: "9c6a93ea324e4afc960e099ff730e232" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37715 } } peers { permanent_uuid: "c72c1d866ac34aa3bbcae200c28f8eb3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36695 } }
I20260928 05:17:13.280728 2018 raft_consensus.cc:374] T b9a7de9525da425e88427a6fe8793530 P 757e3e76bff144399dc79b6855581575 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:17:13.280823 2018 raft_consensus.cc:729] T b9a7de9525da425e88427a6fe8793530 P 757e3e76bff144399dc79b6855581575 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 757e3e76bff144399dc79b6855581575, State: Initialized, Role: FOLLOWER
I20260928 05:17:13.281421 2018 consensus_queue.cc:260] T b9a7de9525da425e88427a6fe8793530 P 757e3e76bff144399dc79b6855581575 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "757e3e76bff144399dc79b6855581575" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43191 } } peers { permanent_uuid: "9c6a93ea324e4afc960e099ff730e232" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37715 } } peers { permanent_uuid: "c72c1d866ac34aa3bbcae200c28f8eb3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36695 } }
I20260928 05:17:13.282203 2016 ts_tablet_manager.cc:1428] T b9a7de9525da425e88427a6fe8793530 P 9c6a93ea324e4afc960e099ff730e232: Time spent starting tablet: real 0.015s user 0.013s sys 0.003s
I20260928 05:17:13.283808 1997 heartbeater.cc:503] Master 127.0.97.62:39279 was elected leader, sending a full tablet report...
I20260928 05:17:13.284101 2018 ts_tablet_manager.cc:1428] T b9a7de9525da425e88427a6fe8793530 P 757e3e76bff144399dc79b6855581575: Time spent starting tablet: real 0.013s user 0.010s sys 0.004s
I20260928 05:17:13.353917 2022 raft_consensus.cc:482] T b9a7de9525da425e88427a6fe8793530 P c72c1d866ac34aa3bbcae200c28f8eb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:17:13.354264 2022 raft_consensus.cc:504] T b9a7de9525da425e88427a6fe8793530 P c72c1d866ac34aa3bbcae200c28f8eb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "757e3e76bff144399dc79b6855581575" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43191 } } peers { permanent_uuid: "9c6a93ea324e4afc960e099ff730e232" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37715 } } peers { permanent_uuid: "c72c1d866ac34aa3bbcae200c28f8eb3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36695 } }
I20260928 05:17:13.355851 2022 leader_election.cc:288] T b9a7de9525da425e88427a6fe8793530 P c72c1d866ac34aa3bbcae200c28f8eb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 757e3e76bff144399dc79b6855581575 (127.0.97.3:43191), 9c6a93ea324e4afc960e099ff730e232 (127.0.97.1:37715)
W20260928 05:17:13.361457 1719 tablet.cc:2369] T b9a7de9525da425e88427a6fe8793530 P 9c6a93ea324e4afc960e099ff730e232: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 05:17:13.362660 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:13.356182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42171 (local address 127.0.97.3:43191)
0928 05:17:13.356374 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:13.356381 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:13.357715 (+ 1334us) server_negotiation.cc:407] Connection header received
0928 05:17:13.357898 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:13.357902 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:13.357962 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:13.358067 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:17:13.358598 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:13.359280 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:13.361029 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:13.361832 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:13.361941 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:13.362050 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:13.362329 (+ 279us) server_negotiation.cc:299] Negotiation successful
0928 05:17:13.362452 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 05:17:13.363052 2025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:13.356262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43191 (local address 127.0.97.2:42171)
0928 05:17:13.357617 (+ 1355us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:13.357644 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:13.357756 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:13.358100 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:13.358104 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:13.358133 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:17:13.358457 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:13.358465 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:13.359405 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:13.359410 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:13.360904 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:13.360915 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:13.362020 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:13.362026 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:13.362176 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:13.362216 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 05:17:13.362300 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 05:17:13.362425 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1232,"mutex_wait_us":3,"spinlock_wait_cycles":2304,"thread_start_us":395,"threads_started":1}
I20260928 05:17:13.363075 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:13.356224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40455 (local address 127.0.97.1:37715)
0928 05:17:13.356407 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:13.356413 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:13.357607 (+ 1194us) server_negotiation.cc:407] Connection header received
0928 05:17:13.357807 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:13.357812 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:13.357897 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:13.357969 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:17:13.358587 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:13.359277 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:13.361066 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:13.362181 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:13.362322 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:13.362410 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:13.362800 (+ 390us) server_negotiation.cc:299] Negotiation successful
0928 05:17:13.362925 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 05:17:13.363286 2026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:13.356224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37715 (local address 127.0.97.2:40455)
0928 05:17:13.357503 (+ 1279us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:13.357518 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:13.357651 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:13.358003 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:13.358008 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:13.358030 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:17:13.358445 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:13.358455 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:13.359398 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:13.359403 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:13.360918 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:13.360931 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:13.362427 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:13.362432 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:13.362632 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:13.362661 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 05:17:13.362745 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 05:17:13.362890 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1175,"mutex_wait_us":20,"spinlock_wait_cycles":896,"thread_start_us":472,"threads_started":1}
I20260928 05:17:13.364068 1952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9a7de9525da425e88427a6fe8793530" candidate_uuid: "c72c1d866ac34aa3bbcae200c28f8eb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "757e3e76bff144399dc79b6855581575" is_pre_election: true
I20260928 05:17:13.364157 1673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9a7de9525da425e88427a6fe8793530" candidate_uuid: "c72c1d866ac34aa3bbcae200c28f8eb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c6a93ea324e4afc960e099ff730e232" is_pre_election: true
I20260928 05:17:13.364513 1952 raft_consensus.cc:2454] T b9a7de9525da425e88427a6fe8793530 P 757e3e76bff144399dc79b6855581575 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c72c1d866ac34aa3bbcae200c28f8eb3 in term 0.
I20260928 05:17:13.364588 1673 raft_consensus.cc:2454] T b9a7de9525da425e88427a6fe8793530 P 9c6a93ea324e4afc960e099ff730e232 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c72c1d866ac34aa3bbcae200c28f8eb3 in term 0.
I20260928 05:17:13.365185 1739 leader_election.cc:302] T b9a7de9525da425e88427a6fe8793530 P c72c1d866ac34aa3bbcae200c28f8eb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 757e3e76bff144399dc79b6855581575, 9c6a93ea324e4afc960e099ff730e232, c72c1d866ac34aa3bbcae200c28f8eb3; no voters:
I20260928 05:17:13.365602 2022 raft_consensus.cc:2781] T b9a7de9525da425e88427a6fe8793530 P c72c1d866ac34aa3bbcae200c28f8eb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 05:17:13.365744 2022 raft_consensus.cc:482] T b9a7de9525da425e88427a6fe8793530 P c72c1d866ac34aa3bbcae200c28f8eb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 05:17:13.365842 2022 raft_consensus.cc:3037] T b9a7de9525da425e88427a6fe8793530 P c72c1d866ac34aa3bbcae200c28f8eb3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:17:13.368255 2022 raft_consensus.cc:504] T b9a7de9525da425e88427a6fe8793530 P c72c1d866ac34aa3bbcae200c28f8eb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "757e3e76bff144399dc79b6855581575" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43191 } } peers { permanent_uuid: "9c6a93ea324e4afc960e099ff730e232" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37715 } } peers { permanent_uuid: "c72c1d866ac34aa3bbcae200c28f8eb3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36695 } }
I20260928 05:17:13.368789 2022 leader_election.cc:288] T b9a7de9525da425e88427a6fe8793530 P c72c1d866ac34aa3bbcae200c28f8eb3 [CANDIDATE]: Term 1 election: Requested vote from peers 757e3e76bff144399dc79b6855581575 (127.0.97.3:43191), 9c6a93ea324e4afc960e099ff730e232 (127.0.97.1:37715)
I20260928 05:17:13.369230 1673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9a7de9525da425e88427a6fe8793530" candidate_uuid: "c72c1d866ac34aa3bbcae200c28f8eb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c6a93ea324e4afc960e099ff730e232"
I20260928 05:17:13.369213 1952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9a7de9525da425e88427a6fe8793530" candidate_uuid: "c72c1d866ac34aa3bbcae200c28f8eb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "757e3e76bff144399dc79b6855581575"
I20260928 05:17:13.369396 1673 raft_consensus.cc:3037] T b9a7de9525da425e88427a6fe8793530 P 9c6a93ea324e4afc960e099ff730e232 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:17:13.369397 1952 raft_consensus.cc:3037] T b9a7de9525da425e88427a6fe8793530 P 757e3e76bff144399dc79b6855581575 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:17:13.371447 1673 raft_consensus.cc:2454] T b9a7de9525da425e88427a6fe8793530 P 9c6a93ea324e4afc960e099ff730e232 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c72c1d866ac34aa3bbcae200c28f8eb3 in term 1.
I20260928 05:17:13.371482 1952 raft_consensus.cc:2454] T b9a7de9525da425e88427a6fe8793530 P 757e3e76bff144399dc79b6855581575 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c72c1d866ac34aa3bbcae200c28f8eb3 in term 1.
I20260928 05:17:13.371810 1739 leader_election.cc:302] T b9a7de9525da425e88427a6fe8793530 P c72c1d866ac34aa3bbcae200c28f8eb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 757e3e76bff144399dc79b6855581575, 9c6a93ea324e4afc960e099ff730e232, c72c1d866ac34aa3bbcae200c28f8eb3; no voters:
I20260928 05:17:13.372045 2022 raft_consensus.cc:2781] T b9a7de9525da425e88427a6fe8793530 P c72c1d866ac34aa3bbcae200c28f8eb3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:17:13.372422 2022 raft_consensus.cc:686] T b9a7de9525da425e88427a6fe8793530 P c72c1d866ac34aa3bbcae200c28f8eb3 [term 1 LEADER]: Becoming Leader. State: Replica: c72c1d866ac34aa3bbcae200c28f8eb3, State: Running, Role: LEADER
I20260928 05:17:13.372905 2022 consensus_queue.cc:237] T b9a7de9525da425e88427a6fe8793530 P c72c1d866ac34aa3bbcae200c28f8eb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "757e3e76bff144399dc79b6855581575" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43191 } } peers { permanent_uuid: "9c6a93ea324e4afc960e099ff730e232" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37715 } } peers { permanent_uuid: "c72c1d866ac34aa3bbcae200c28f8eb3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36695 } }
I20260928 05:17:13.376868 1510 catalog_manager.cc:5495] T b9a7de9525da425e88427a6fe8793530 P c72c1d866ac34aa3bbcae200c28f8eb3 reported cstate change: term changed from 0 to 1, leader changed from <none> to c72c1d866ac34aa3bbcae200c28f8eb3 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "c72c1d866ac34aa3bbcae200c28f8eb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "757e3e76bff144399dc79b6855581575" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c6a93ea324e4afc960e099ff730e232" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37715 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c72c1d866ac34aa3bbcae200c28f8eb3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36695 } health_report { overall_health: HEALTHY } } }
W20260928 05:17:13.402390 1998 tablet.cc:2369] T b9a7de9525da425e88427a6fe8793530 P 757e3e76bff144399dc79b6855581575: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 05:17:13.423173 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:13.417099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53498 (local address 127.0.97.2:36695)
0928 05:17:13.417279 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:13.417285 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:13.417402 (+ 117us) server_negotiation.cc:407] Connection header received
0928 05:17:13.417696 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:13.417700 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:13.417769 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:13.417882 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:17:13.418678 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:13.419591 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:13.421034 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:13.421446 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:13.421564 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:13.422051 (+ 487us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:17:13.422560 (+ 509us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:17:13.422689 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:13.422825 (+ 136us) server_negotiation.cc:299] Negotiation successful
0928 05:17:13.422915 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 05:17:13.430552 1673 raft_consensus.cc:1261] T b9a7de9525da425e88427a6fe8793530 P 9c6a93ea324e4afc960e099ff730e232 [term 1 FOLLOWER]: Refusing update from remote peer c72c1d866ac34aa3bbcae200c28f8eb3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:17:13.430550 1952 raft_consensus.cc:1261] T b9a7de9525da425e88427a6fe8793530 P 757e3e76bff144399dc79b6855581575 [term 1 FOLLOWER]: Refusing update from remote peer c72c1d866ac34aa3bbcae200c28f8eb3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:17:13.431485 2027 consensus_queue.cc:1035] T b9a7de9525da425e88427a6fe8793530 P c72c1d866ac34aa3bbcae200c28f8eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c6a93ea324e4afc960e099ff730e232" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:17:13.431813 2022 consensus_queue.cc:1035] T b9a7de9525da425e88427a6fe8793530 P c72c1d866ac34aa3bbcae200c28f8eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "757e3e76bff144399dc79b6855581575" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 05:17:13.436167 1859 tablet.cc:2369] T b9a7de9525da425e88427a6fe8793530 P c72c1d866ac34aa3bbcae200c28f8eb3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 05:17:13.446462 2031 mvcc.cc:205] Tried to move back new op lower bound from 7334185506522050560 to 7334185506298839040. Current Snapshot: MvccSnapshot[applied={T|T < 7334185506522050560}]
I20260928 05:17:13.458192 1510 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55514:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:17:13.458385 1510 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55514:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:17:13.461448 1510 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d09101969d894158a93c73549cbd740f: Sending DeleteTablet for 3 replicas of tablet b9a7de9525da425e88427a6fe8793530
I20260928 05:17:13.462385 1932 tablet_service.cc:1514] Processing DeleteTablet for tablet b9a7de9525da425e88427a6fe8793530 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:17:13 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54108
I20260928 05:17:13.462491 1648 tablet_service.cc:1514] Processing DeleteTablet for tablet b9a7de9525da425e88427a6fe8793530 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:17:13 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41062
I20260928 05:17:13.463344 1788 tablet_service.cc:1514] Processing DeleteTablet for tablet b9a7de9525da425e88427a6fe8793530 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:17:13 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53492
I20260928 05:17:13.463474 1510 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55514:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 05:17:13.464071 1510 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 05:17:13.469096 2040 tablet_replica.cc:330] T b9a7de9525da425e88427a6fe8793530 P 9c6a93ea324e4afc960e099ff730e232: stopping tablet replica
I20260928 05:17:13.469551 2040 raft_consensus.cc:2229] T b9a7de9525da425e88427a6fe8793530 P 9c6a93ea324e4afc960e099ff730e232 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:17:13.470058 2040 raft_consensus.cc:2258] T b9a7de9525da425e88427a6fe8793530 P 9c6a93ea324e4afc960e099ff730e232 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:17:13.471954 2040 ts_tablet_manager.cc:1905] T b9a7de9525da425e88427a6fe8793530 P 9c6a93ea324e4afc960e099ff730e232: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:17:13.472007 1932 tablet_service.cc:1467] Processing CreateTablet for tablet e9efd5f6a9f44a0b929a36705f29f50b (DEFAULT_TABLE table=test-table [id=2c6d1403a7874789a4be1eaa0cfd44b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:17:13.472460 1932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9efd5f6a9f44a0b929a36705f29f50b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:17:13.477268 1787 tablet_service.cc:1467] Processing CreateTablet for tablet e9efd5f6a9f44a0b929a36705f29f50b (DEFAULT_TABLE table=test-table [id=2c6d1403a7874789a4be1eaa0cfd44b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:17:13.477857 1787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9efd5f6a9f44a0b929a36705f29f50b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:17:13.478521 2041 tablet_replica.cc:330] T b9a7de9525da425e88427a6fe8793530 P c72c1d866ac34aa3bbcae200c28f8eb3: stopping tablet replica
I20260928 05:17:13.478921 2041 raft_consensus.cc:2229] T b9a7de9525da425e88427a6fe8793530 P c72c1d866ac34aa3bbcae200c28f8eb3 [term 1 LEADER]: Raft consensus shutting down.
I20260928 05:17:13.479540 2041 raft_consensus.cc:2258] T b9a7de9525da425e88427a6fe8793530 P c72c1d866ac34aa3bbcae200c28f8eb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:17:13.479722 2040 ts_tablet_manager.cc:1918] T b9a7de9525da425e88427a6fe8793530 P 9c6a93ea324e4afc960e099ff730e232: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:17:13.479905 2040 log.cc:1196] T b9a7de9525da425e88427a6fe8793530 P 9c6a93ea324e4afc960e099ff730e232: Deleting WAL directory at /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/minicluster-data/ts-0/wal/wals/b9a7de9525da425e88427a6fe8793530
I20260928 05:17:13.480441 2040 ts_tablet_manager.cc:1939] T b9a7de9525da425e88427a6fe8793530 P 9c6a93ea324e4afc960e099ff730e232: Deleting consensus metadata
I20260928 05:17:13.481380 2041 ts_tablet_manager.cc:1905] T b9a7de9525da425e88427a6fe8793530 P c72c1d866ac34aa3bbcae200c28f8eb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:17:13.481990 1648 tablet_service.cc:1467] Processing CreateTablet for tablet e9efd5f6a9f44a0b929a36705f29f50b (DEFAULT_TABLE table=test-table [id=2c6d1403a7874789a4be1eaa0cfd44b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:17:13.482362 1648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9efd5f6a9f44a0b929a36705f29f50b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:17:13.484149 1496 catalog_manager.cc:4841] TS 9c6a93ea324e4afc960e099ff730e232 (127.0.97.1:37715): tablet b9a7de9525da425e88427a6fe8793530 (table test-table [id=4d5336f4ba314fc5b180312e7da24187]) successfully deleted
I20260928 05:17:13.488457 2041 ts_tablet_manager.cc:1918] T b9a7de9525da425e88427a6fe8793530 P c72c1d866ac34aa3bbcae200c28f8eb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:17:13.488615 2041 log.cc:1196] T b9a7de9525da425e88427a6fe8793530 P c72c1d866ac34aa3bbcae200c28f8eb3: Deleting WAL directory at /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/minicluster-data/ts-1/wal/wals/b9a7de9525da425e88427a6fe8793530
I20260928 05:17:13.489051 2017 tablet_bootstrap.cc:492] T e9efd5f6a9f44a0b929a36705f29f50b P c72c1d866ac34aa3bbcae200c28f8eb3: Bootstrap starting.
I20260928 05:17:13.489184 2041 ts_tablet_manager.cc:1939] T b9a7de9525da425e88427a6fe8793530 P c72c1d866ac34aa3bbcae200c28f8eb3: Deleting consensus metadata
I20260928 05:17:13.490651 2017 tablet_bootstrap.cc:654] T e9efd5f6a9f44a0b929a36705f29f50b P c72c1d866ac34aa3bbcae200c28f8eb3: Neither blocks nor log segments found. Creating new log.
I20260928 05:17:13.491549 1496 catalog_manager.cc:4841] TS c72c1d866ac34aa3bbcae200c28f8eb3 (127.0.97.2:36695): tablet b9a7de9525da425e88427a6fe8793530 (table test-table [id=4d5336f4ba314fc5b180312e7da24187]) successfully deleted
I20260928 05:17:13.492709 2017 tablet_bootstrap.cc:492] T e9efd5f6a9f44a0b929a36705f29f50b P c72c1d866ac34aa3bbcae200c28f8eb3: No bootstrap required, opened a new log
I20260928 05:17:13.492826 2017 ts_tablet_manager.cc:1397] T e9efd5f6a9f44a0b929a36705f29f50b P c72c1d866ac34aa3bbcae200c28f8eb3: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260928 05:17:13.493667 2039 tablet_replica.cc:330] T b9a7de9525da425e88427a6fe8793530 P 757e3e76bff144399dc79b6855581575: stopping tablet replica
I20260928 05:17:13.493717 2018 tablet_bootstrap.cc:492] T e9efd5f6a9f44a0b929a36705f29f50b P 757e3e76bff144399dc79b6855581575: Bootstrap starting.
I20260928 05:17:13.493901 2017 raft_consensus.cc:348] T e9efd5f6a9f44a0b929a36705f29f50b P c72c1d866ac34aa3bbcae200c28f8eb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c6a93ea324e4afc960e099ff730e232" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37715 } } peers { permanent_uuid: "c72c1d866ac34aa3bbcae200c28f8eb3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36695 } } peers { permanent_uuid: "757e3e76bff144399dc79b6855581575" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43191 } }
I20260928 05:17:13.494097 2017 raft_consensus.cc:374] T e9efd5f6a9f44a0b929a36705f29f50b P c72c1d866ac34aa3bbcae200c28f8eb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:17:13.494194 2017 raft_consensus.cc:729] T e9efd5f6a9f44a0b929a36705f29f50b P c72c1d866ac34aa3bbcae200c28f8eb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c72c1d866ac34aa3bbcae200c28f8eb3, State: Initialized, Role: FOLLOWER
I20260928 05:17:13.494375 2039 raft_consensus.cc:2229] T b9a7de9525da425e88427a6fe8793530 P 757e3e76bff144399dc79b6855581575 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:17:13.494442 2017 consensus_queue.cc:260] T e9efd5f6a9f44a0b929a36705f29f50b P c72c1d866ac34aa3bbcae200c28f8eb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c6a93ea324e4afc960e099ff730e232" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37715 } } peers { permanent_uuid: "c72c1d866ac34aa3bbcae200c28f8eb3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36695 } } peers { permanent_uuid: "757e3e76bff144399dc79b6855581575" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43191 } }
I20260928 05:17:13.495457 2039 raft_consensus.cc:2258] T b9a7de9525da425e88427a6fe8793530 P 757e3e76bff144399dc79b6855581575 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:17:13.495875 2018 tablet_bootstrap.cc:654] T e9efd5f6a9f44a0b929a36705f29f50b P 757e3e76bff144399dc79b6855581575: Neither blocks nor log segments found. Creating new log.
I20260928 05:17:13.496125 2016 tablet_bootstrap.cc:492] T e9efd5f6a9f44a0b929a36705f29f50b P 9c6a93ea324e4afc960e099ff730e232: Bootstrap starting.
I20260928 05:17:13.497540 2017 ts_tablet_manager.cc:1428] T e9efd5f6a9f44a0b929a36705f29f50b P c72c1d866ac34aa3bbcae200c28f8eb3: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260928 05:17:13.497627 2016 tablet_bootstrap.cc:654] T e9efd5f6a9f44a0b929a36705f29f50b P 9c6a93ea324e4afc960e099ff730e232: Neither blocks nor log segments found. Creating new log.
I20260928 05:17:13.499480 2016 tablet_bootstrap.cc:492] T e9efd5f6a9f44a0b929a36705f29f50b P 9c6a93ea324e4afc960e099ff730e232: No bootstrap required, opened a new log
I20260928 05:17:13.499850 2016 ts_tablet_manager.cc:1397] T e9efd5f6a9f44a0b929a36705f29f50b P 9c6a93ea324e4afc960e099ff730e232: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260928 05:17:13.499897 2039 ts_tablet_manager.cc:1905] T b9a7de9525da425e88427a6fe8793530 P 757e3e76bff144399dc79b6855581575: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:17:13.500583 2016 raft_consensus.cc:348] T e9efd5f6a9f44a0b929a36705f29f50b P 9c6a93ea324e4afc960e099ff730e232 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c6a93ea324e4afc960e099ff730e232" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37715 } } peers { permanent_uuid: "c72c1d866ac34aa3bbcae200c28f8eb3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36695 } } peers { permanent_uuid: "757e3e76bff144399dc79b6855581575" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43191 } }
I20260928 05:17:13.500785 2016 raft_consensus.cc:374] T e9efd5f6a9f44a0b929a36705f29f50b P 9c6a93ea324e4afc960e099ff730e232 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:17:13.500902 2016 raft_consensus.cc:729] T e9efd5f6a9f44a0b929a36705f29f50b P 9c6a93ea324e4afc960e099ff730e232 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c6a93ea324e4afc960e099ff730e232, State: Initialized, Role: FOLLOWER
I20260928 05:17:13.501150 2016 consensus_queue.cc:260] T e9efd5f6a9f44a0b929a36705f29f50b P 9c6a93ea324e4afc960e099ff730e232 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c6a93ea324e4afc960e099ff730e232" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37715 } } peers { permanent_uuid: "c72c1d866ac34aa3bbcae200c28f8eb3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36695 } } peers { permanent_uuid: "757e3e76bff144399dc79b6855581575" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43191 } }
I20260928 05:17:13.502112 2016 ts_tablet_manager.cc:1428] T e9efd5f6a9f44a0b929a36705f29f50b P 9c6a93ea324e4afc960e099ff730e232: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 05:17:13.503142 2018 tablet_bootstrap.cc:492] T e9efd5f6a9f44a0b929a36705f29f50b P 757e3e76bff144399dc79b6855581575: No bootstrap required, opened a new log
I20260928 05:17:13.503269 2018 ts_tablet_manager.cc:1397] T e9efd5f6a9f44a0b929a36705f29f50b P 757e3e76bff144399dc79b6855581575: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.004s
I20260928 05:17:13.503991 2018 raft_consensus.cc:348] T e9efd5f6a9f44a0b929a36705f29f50b P 757e3e76bff144399dc79b6855581575 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c6a93ea324e4afc960e099ff730e232" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37715 } } peers { permanent_uuid: "c72c1d866ac34aa3bbcae200c28f8eb3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36695 } } peers { permanent_uuid: "757e3e76bff144399dc79b6855581575" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43191 } }
I20260928 05:17:13.504205 2018 raft_consensus.cc:374] T e9efd5f6a9f44a0b929a36705f29f50b P 757e3e76bff144399dc79b6855581575 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:17:13.504308 2018 raft_consensus.cc:729] T e9efd5f6a9f44a0b929a36705f29f50b P 757e3e76bff144399dc79b6855581575 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 757e3e76bff144399dc79b6855581575, State: Initialized, Role: FOLLOWER
I20260928 05:17:13.504568 2018 consensus_queue.cc:260] T e9efd5f6a9f44a0b929a36705f29f50b P 757e3e76bff144399dc79b6855581575 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c6a93ea324e4afc960e099ff730e232" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37715 } } peers { permanent_uuid: "c72c1d866ac34aa3bbcae200c28f8eb3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36695 } } peers { permanent_uuid: "757e3e76bff144399dc79b6855581575" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43191 } }
I20260928 05:17:13.505405 2018 ts_tablet_manager.cc:1428] T e9efd5f6a9f44a0b929a36705f29f50b P 757e3e76bff144399dc79b6855581575: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260928 05:17:13.507463 2039 ts_tablet_manager.cc:1918] T b9a7de9525da425e88427a6fe8793530 P 757e3e76bff144399dc79b6855581575: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:17:13.507611 2039 log.cc:1196] T b9a7de9525da425e88427a6fe8793530 P 757e3e76bff144399dc79b6855581575: Deleting WAL directory at /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/minicluster-data/ts-2/wal/wals/b9a7de9525da425e88427a6fe8793530
I20260928 05:17:13.508069 2039 ts_tablet_manager.cc:1939] T b9a7de9525da425e88427a6fe8793530 P 757e3e76bff144399dc79b6855581575: Deleting consensus metadata
I20260928 05:17:13.509660 1495 catalog_manager.cc:4841] TS 757e3e76bff144399dc79b6855581575 (127.0.97.3:43191): tablet b9a7de9525da425e88427a6fe8793530 (table test-table [id=4d5336f4ba314fc5b180312e7da24187]) successfully deleted
I20260928 05:17:13.672040 2024 raft_consensus.cc:482] T e9efd5f6a9f44a0b929a36705f29f50b P 757e3e76bff144399dc79b6855581575 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:17:13.672230 2024 raft_consensus.cc:504] T e9efd5f6a9f44a0b929a36705f29f50b P 757e3e76bff144399dc79b6855581575 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c6a93ea324e4afc960e099ff730e232" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37715 } } peers { permanent_uuid: "c72c1d866ac34aa3bbcae200c28f8eb3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36695 } } peers { permanent_uuid: "757e3e76bff144399dc79b6855581575" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43191 } }
I20260928 05:17:13.673599 2024 leader_election.cc:288] T e9efd5f6a9f44a0b929a36705f29f50b P 757e3e76bff144399dc79b6855581575 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c6a93ea324e4afc960e099ff730e232 (127.0.97.1:37715), c72c1d866ac34aa3bbcae200c28f8eb3 (127.0.97.2:36695)
I20260928 05:17:13.680510 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:13.674043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36505 (local address 127.0.97.1:37715)
0928 05:17:13.674223 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:13.674229 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:13.674392 (+ 163us) server_negotiation.cc:407] Connection header received
0928 05:17:13.674684 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:13.674690 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:13.674784 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:13.674849 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:17:13.675394 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:13.676313 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:13.678599 (+ 2286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:13.679624 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:13.679735 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:13.679834 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:13.680113 (+ 279us) server_negotiation.cc:299] Negotiation successful
0928 05:17:13.680244 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 05:17:13.680651 2045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:13.673849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37715 (local address 127.0.97.3:36505)
0928 05:17:13.674303 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:13.674321 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:13.674508 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:13.675031 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:13.675034 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:13.675058 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:17:13.675249 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:13.675257 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:13.676447 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:13.676451 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:13.678433 (+ 1982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:13.678447 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:13.679767 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:13.679773 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:13.679942 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:13.679973 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 05:17:13.680059 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 05:17:13.680249 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"mutex_wait_us":1,"thread_start_us":241,"threads_started":1}
I20260928 05:17:13.681264 1673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9efd5f6a9f44a0b929a36705f29f50b" candidate_uuid: "757e3e76bff144399dc79b6855581575" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c6a93ea324e4afc960e099ff730e232" is_pre_election: true
I20260928 05:17:13.681484 1673 raft_consensus.cc:2454] T e9efd5f6a9f44a0b929a36705f29f50b P 9c6a93ea324e4afc960e099ff730e232 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 757e3e76bff144399dc79b6855581575 in term 0.
I20260928 05:17:13.682037 2046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:13.674439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36695 (local address 127.0.97.3:43845)
0928 05:17:13.675252 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:13.675280 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:13.675526 (+ 246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:13.675917 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:13.675921 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:13.675947 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:17:13.676240 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:13.676251 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:13.677367 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:13.677373 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:13.678986 (+ 1613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:13.679000 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:13.681242 (+ 2242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:13.681248 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:13.681427 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:13.681470 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 05:17:13.681560 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 05:17:13.681695 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":679,"thread_start_us":391,"threads_started":1}
I20260928 05:17:13.682330 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:13.674562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43845 (local address 127.0.97.2:36695)
0928 05:17:13.674723 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:13.674728 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:13.675394 (+ 666us) server_negotiation.cc:407] Connection header received
0928 05:17:13.675698 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:13.675703 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:13.675801 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:13.675889 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:17:13.676407 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:13.677258 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:13.679163 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:13.680556 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:13.680693 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:13.680788 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:13.681646 (+ 858us) server_negotiation.cc:299] Negotiation successful
0928 05:17:13.681774 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 05:17:13.682399 1879 leader_election.cc:302] T e9efd5f6a9f44a0b929a36705f29f50b P 757e3e76bff144399dc79b6855581575 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 757e3e76bff144399dc79b6855581575, 9c6a93ea324e4afc960e099ff730e232; no voters:
I20260928 05:17:13.682870 2024 raft_consensus.cc:2781] T e9efd5f6a9f44a0b929a36705f29f50b P 757e3e76bff144399dc79b6855581575 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 05:17:13.682976 2024 raft_consensus.cc:482] T e9efd5f6a9f44a0b929a36705f29f50b P 757e3e76bff144399dc79b6855581575 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 05:17:13.683061 2024 raft_consensus.cc:3037] T e9efd5f6a9f44a0b929a36705f29f50b P 757e3e76bff144399dc79b6855581575 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:17:13.685185 1808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9efd5f6a9f44a0b929a36705f29f50b" candidate_uuid: "757e3e76bff144399dc79b6855581575" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c72c1d866ac34aa3bbcae200c28f8eb3" is_pre_election: true
I20260928 05:17:13.685580 1808 raft_consensus.cc:2454] T e9efd5f6a9f44a0b929a36705f29f50b P c72c1d866ac34aa3bbcae200c28f8eb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 757e3e76bff144399dc79b6855581575 in term 0.
I20260928 05:17:13.686120 2024 raft_consensus.cc:504] T e9efd5f6a9f44a0b929a36705f29f50b P 757e3e76bff144399dc79b6855581575 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c6a93ea324e4afc960e099ff730e232" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37715 } } peers { permanent_uuid: "c72c1d866ac34aa3bbcae200c28f8eb3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36695 } } peers { permanent_uuid: "757e3e76bff144399dc79b6855581575" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43191 } }
I20260928 05:17:13.686832 2024 leader_election.cc:288] T e9efd5f6a9f44a0b929a36705f29f50b P 757e3e76bff144399dc79b6855581575 [CANDIDATE]: Term 1 election: Requested vote from peers 9c6a93ea324e4afc960e099ff730e232 (127.0.97.1:37715), c72c1d866ac34aa3bbcae200c28f8eb3 (127.0.97.2:36695)
I20260928 05:17:13.687532 1673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9efd5f6a9f44a0b929a36705f29f50b" candidate_uuid: "757e3e76bff144399dc79b6855581575" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c6a93ea324e4afc960e099ff730e232"
I20260928 05:17:13.687712 1673 raft_consensus.cc:3037] T e9efd5f6a9f44a0b929a36705f29f50b P 9c6a93ea324e4afc960e099ff730e232 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:17:13.689110 1808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9efd5f6a9f44a0b929a36705f29f50b" candidate_uuid: "757e3e76bff144399dc79b6855581575" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c72c1d866ac34aa3bbcae200c28f8eb3"
I20260928 05:17:13.689299 1808 raft_consensus.cc:3037] T e9efd5f6a9f44a0b929a36705f29f50b P c72c1d866ac34aa3bbcae200c28f8eb3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:17:13.690497 1673 raft_consensus.cc:2454] T e9efd5f6a9f44a0b929a36705f29f50b P 9c6a93ea324e4afc960e099ff730e232 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 757e3e76bff144399dc79b6855581575 in term 1.
I20260928 05:17:13.690886 1879 leader_election.cc:302] T e9efd5f6a9f44a0b929a36705f29f50b P 757e3e76bff144399dc79b6855581575 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 757e3e76bff144399dc79b6855581575, 9c6a93ea324e4afc960e099ff730e232; no voters:
I20260928 05:17:13.691211 2024 raft_consensus.cc:2781] T e9efd5f6a9f44a0b929a36705f29f50b P 757e3e76bff144399dc79b6855581575 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:17:13.692725 1808 raft_consensus.cc:2454] T e9efd5f6a9f44a0b929a36705f29f50b P c72c1d866ac34aa3bbcae200c28f8eb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 757e3e76bff144399dc79b6855581575 in term 1.
I20260928 05:17:13.693534 2024 raft_consensus.cc:686] T e9efd5f6a9f44a0b929a36705f29f50b P 757e3e76bff144399dc79b6855581575 [term 1 LEADER]: Becoming Leader. State: Replica: 757e3e76bff144399dc79b6855581575, State: Running, Role: LEADER
I20260928 05:17:13.694140 2024 consensus_queue.cc:237] T e9efd5f6a9f44a0b929a36705f29f50b P 757e3e76bff144399dc79b6855581575 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c6a93ea324e4afc960e099ff730e232" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37715 } } peers { permanent_uuid: "c72c1d866ac34aa3bbcae200c28f8eb3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36695 } } peers { permanent_uuid: "757e3e76bff144399dc79b6855581575" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43191 } }
I20260928 05:17:13.698885 1511 catalog_manager.cc:5495] T e9efd5f6a9f44a0b929a36705f29f50b P 757e3e76bff144399dc79b6855581575 reported cstate change: term changed from 0 to 1, leader changed from <none> to 757e3e76bff144399dc79b6855581575 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "757e3e76bff144399dc79b6855581575" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c6a93ea324e4afc960e099ff730e232" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37715 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c72c1d866ac34aa3bbcae200c28f8eb3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "757e3e76bff144399dc79b6855581575" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43191 } health_report { overall_health: HEALTHY } } }
Sep 28 05:17:13 dist-test-slave-whsn krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572631, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:17:13.755612 2054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:13.742942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55520 (local address 127.0.97.62:39279)
0928 05:17:13.743372 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:13.743382 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:13.743709 (+ 327us) server_negotiation.cc:407] Connection header received
0928 05:17:13.743881 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:13.743884 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:13.743939 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:13.744022 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:13.744950 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:13.745643 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:13.746763 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:13.747142 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:13.747240 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:13.750413 (+ 3173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:13.750417 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:13.750420 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:13.750447 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:13.752737 (+ 2290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:13.753627 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:13.753632 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:13.753635 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:13.753778 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:13.754160 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:13.754163 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:13.754164 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:13.754391 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:13.754487 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:13.754964 (+ 477us) server_negotiation.cc:299] Negotiation successful
0928 05:17:13.755296 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":121,"server-negotiator.queue_time_us":308,"thread_start_us":204,"threads_started":1}
I20260928 05:17:13.755656 2055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:13.742905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39279 (local address 127.0.0.1:55520)
0928 05:17:13.743597 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:13.743616 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:13.743761 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:13.744061 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:13.744065 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:13.744646 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:17:13.744818 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:13.744827 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:13.745773 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:13.745778 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:13.746639 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:13.746649 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:13.747267 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:13.747273 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:13.747431 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:13.747986 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:17:13.748002 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:17:13.750239 (+ 2237us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:17:13.752890 (+ 2651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:13.752902 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:13.752910 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:13.753509 (+ 599us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:13.753877 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:13.753882 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:13.753886 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:13.754058 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:13.754527 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:17:13.754535 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:17:13.754717 (+ 182us) client_negotiation.cc:769] Sending connection context
0928 05:17:13.754889 (+ 172us) client_negotiation.cc:240] Negotiation successful
0928 05:17:13.755196 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":433,"threads_started":1}
I20260928 05:17:13.760535 1511 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55520:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 05:17:13.761579 1511 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 05:17:13.768535 1648 tablet_service.cc:1467] Processing CreateTablet for tablet e5935a6a6ab44823995e601ea733c89d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=96cc33a2712048d99b0ceb4b3fdae6ef]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 05:17:13.768666 1787 tablet_service.cc:1467] Processing CreateTablet for tablet e5935a6a6ab44823995e601ea733c89d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=96cc33a2712048d99b0ceb4b3fdae6ef]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 05:17:13.769106 1648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5935a6a6ab44823995e601ea733c89d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:17:13.769212 1787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5935a6a6ab44823995e601ea733c89d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:17:13.774227 2016 tablet_bootstrap.cc:492] T e5935a6a6ab44823995e601ea733c89d P 9c6a93ea324e4afc960e099ff730e232: Bootstrap starting.
I20260928 05:17:13.774749 1932 tablet_service.cc:1467] Processing CreateTablet for tablet e5935a6a6ab44823995e601ea733c89d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=96cc33a2712048d99b0ceb4b3fdae6ef]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 05:17:13.775357 1932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5935a6a6ab44823995e601ea733c89d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:17:13.776499 2016 tablet_bootstrap.cc:654] T e5935a6a6ab44823995e601ea733c89d P 9c6a93ea324e4afc960e099ff730e232: Neither blocks nor log segments found. Creating new log.
I20260928 05:17:13.777632 2017 tablet_bootstrap.cc:492] T e5935a6a6ab44823995e601ea733c89d P c72c1d866ac34aa3bbcae200c28f8eb3: Bootstrap starting.
I20260928 05:17:13.779413 2017 tablet_bootstrap.cc:654] T e5935a6a6ab44823995e601ea733c89d P c72c1d866ac34aa3bbcae200c28f8eb3: Neither blocks nor log segments found. Creating new log.
I20260928 05:17:13.783241 2018 tablet_bootstrap.cc:492] T e5935a6a6ab44823995e601ea733c89d P 757e3e76bff144399dc79b6855581575: Bootstrap starting.
I20260928 05:17:13.785050 2018 tablet_bootstrap.cc:654] T e5935a6a6ab44823995e601ea733c89d P 757e3e76bff144399dc79b6855581575: Neither blocks nor log segments found. Creating new log.
I20260928 05:17:13.787238 2018 tablet_bootstrap.cc:492] T e5935a6a6ab44823995e601ea733c89d P 757e3e76bff144399dc79b6855581575: No bootstrap required, opened a new log
I20260928 05:17:13.787401 2018 ts_tablet_manager.cc:1397] T e5935a6a6ab44823995e601ea733c89d P 757e3e76bff144399dc79b6855581575: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20260928 05:17:13.788163 2018 raft_consensus.cc:348] T e5935a6a6ab44823995e601ea733c89d P 757e3e76bff144399dc79b6855581575 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c6a93ea324e4afc960e099ff730e232" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37715 } } peers { permanent_uuid: "c72c1d866ac34aa3bbcae200c28f8eb3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36695 } } peers { permanent_uuid: "757e3e76bff144399dc79b6855581575" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43191 } }
I20260928 05:17:13.788416 2018 raft_consensus.cc:374] T e5935a6a6ab44823995e601ea733c89d P 757e3e76bff144399dc79b6855581575 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:17:13.788492 2018 raft_consensus.cc:729] T e5935a6a6ab44823995e601ea733c89d P 757e3e76bff144399dc79b6855581575 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 757e3e76bff144399dc79b6855581575, State: Initialized, Role: FOLLOWER
I20260928 05:17:13.788720 2018 consensus_queue.cc:260] T e5935a6a6ab44823995e601ea733c89d P 757e3e76bff144399dc79b6855581575 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c6a93ea324e4afc960e099ff730e232" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37715 } } peers { permanent_uuid: "c72c1d866ac34aa3bbcae200c28f8eb3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36695 } } peers { permanent_uuid: "757e3e76bff144399dc79b6855581575" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43191 } }
I20260928 05:17:13.789350 2018 ts_tablet_manager.cc:1428] T e5935a6a6ab44823995e601ea733c89d P 757e3e76bff144399dc79b6855581575: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260928 05:17:13.790563 2024 tablet_replica.cc:439] T e5935a6a6ab44823995e601ea733c89d P 757e3e76bff144399dc79b6855581575: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c6a93ea324e4afc960e099ff730e232" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37715 } } peers { permanent_uuid: "c72c1d866ac34aa3bbcae200c28f8eb3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36695 } } peers { permanent_uuid: "757e3e76bff144399dc79b6855581575" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43191 } } }
I20260928 05:17:13.790861 2024 tablet_replica.cc:442] T e5935a6a6ab44823995e601ea733c89d P 757e3e76bff144399dc79b6855581575: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 05:17:13.793599 2016 tablet_bootstrap.cc:492] T e5935a6a6ab44823995e601ea733c89d P 9c6a93ea324e4afc960e099ff730e232: No bootstrap required, opened a new log
I20260928 05:17:13.793777 2016 ts_tablet_manager.cc:1397] T e5935a6a6ab44823995e601ea733c89d P 9c6a93ea324e4afc960e099ff730e232: Time spent bootstrapping tablet: real 0.020s user 0.000s sys 0.009s
I20260928 05:17:13.794044 2017 tablet_bootstrap.cc:492] T e5935a6a6ab44823995e601ea733c89d P c72c1d866ac34aa3bbcae200c28f8eb3: No bootstrap required, opened a new log
I20260928 05:17:13.794214 2017 ts_tablet_manager.cc:1397] T e5935a6a6ab44823995e601ea733c89d P c72c1d866ac34aa3bbcae200c28f8eb3: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.001s
I20260928 05:17:13.794548 2016 raft_consensus.cc:348] T e5935a6a6ab44823995e601ea733c89d P 9c6a93ea324e4afc960e099ff730e232 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c6a93ea324e4afc960e099ff730e232" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37715 } } peers { permanent_uuid: "c72c1d866ac34aa3bbcae200c28f8eb3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36695 } } peers { permanent_uuid: "757e3e76bff144399dc79b6855581575" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43191 } }
I20260928 05:17:13.794792 2016 raft_consensus.cc:374] T e5935a6a6ab44823995e601ea733c89d P 9c6a93ea324e4afc960e099ff730e232 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:17:13.794893 2016 raft_consensus.cc:729] T e5935a6a6ab44823995e601ea733c89d P 9c6a93ea324e4afc960e099ff730e232 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c6a93ea324e4afc960e099ff730e232, State: Initialized, Role: FOLLOWER
I20260928 05:17:13.795028 2017 raft_consensus.cc:348] T e5935a6a6ab44823995e601ea733c89d P c72c1d866ac34aa3bbcae200c28f8eb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c6a93ea324e4afc960e099ff730e232" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37715 } } peers { permanent_uuid: "c72c1d866ac34aa3bbcae200c28f8eb3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36695 } } peers { permanent_uuid: "757e3e76bff144399dc79b6855581575" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43191 } }
I20260928 05:17:13.795275 2017 raft_consensus.cc:374] T e5935a6a6ab44823995e601ea733c89d P c72c1d866ac34aa3bbcae200c28f8eb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:17:13.795138 2016 consensus_queue.cc:260] T e5935a6a6ab44823995e601ea733c89d P 9c6a93ea324e4afc960e099ff730e232 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c6a93ea324e4afc960e099ff730e232" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37715 } } peers { permanent_uuid: "c72c1d866ac34aa3bbcae200c28f8eb3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36695 } } peers { permanent_uuid: "757e3e76bff144399dc79b6855581575" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43191 } }
I20260928 05:17:13.795362 2017 raft_consensus.cc:729] T e5935a6a6ab44823995e601ea733c89d P c72c1d866ac34aa3bbcae200c28f8eb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c72c1d866ac34aa3bbcae200c28f8eb3, State: Initialized, Role: FOLLOWER
I20260928 05:17:13.795595 2023 tablet_replica.cc:439] T e5935a6a6ab44823995e601ea733c89d P 9c6a93ea324e4afc960e099ff730e232: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c6a93ea324e4afc960e099ff730e232" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37715 } } peers { permanent_uuid: "c72c1d866ac34aa3bbcae200c28f8eb3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36695 } } peers { permanent_uuid: "757e3e76bff144399dc79b6855581575" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43191 } } }
I20260928 05:17:13.795980 2023 tablet_replica.cc:442] T e5935a6a6ab44823995e601ea733c89d P 9c6a93ea324e4afc960e099ff730e232: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 05:17:13.796643 2017 consensus_queue.cc:260] T e5935a6a6ab44823995e601ea733c89d P c72c1d866ac34aa3bbcae200c28f8eb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c6a93ea324e4afc960e099ff730e232" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37715 } } peers { permanent_uuid: "c72c1d866ac34aa3bbcae200c28f8eb3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36695 } } peers { permanent_uuid: "757e3e76bff144399dc79b6855581575" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43191 } }
I20260928 05:17:13.797217 2016 ts_tablet_manager.cc:1428] T e5935a6a6ab44823995e601ea733c89d P 9c6a93ea324e4afc960e099ff730e232: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260928 05:17:13.797099 2027 tablet_replica.cc:439] T e5935a6a6ab44823995e601ea733c89d P c72c1d866ac34aa3bbcae200c28f8eb3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c6a93ea324e4afc960e099ff730e232" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37715 } } peers { permanent_uuid: "c72c1d866ac34aa3bbcae200c28f8eb3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36695 } } peers { permanent_uuid: "757e3e76bff144399dc79b6855581575" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43191 } } }
I20260928 05:17:13.797420 2027 tablet_replica.cc:442] T e5935a6a6ab44823995e601ea733c89d P c72c1d866ac34aa3bbcae200c28f8eb3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 05:17:13.799013 2017 ts_tablet_manager.cc:1428] T e5935a6a6ab44823995e601ea733c89d P c72c1d866ac34aa3bbcae200c28f8eb3: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20260928 05:17:13.871883 2024 raft_consensus.cc:482] T e5935a6a6ab44823995e601ea733c89d P 757e3e76bff144399dc79b6855581575 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:17:13.872092 2024 raft_consensus.cc:504] T e5935a6a6ab44823995e601ea733c89d P 757e3e76bff144399dc79b6855581575 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c6a93ea324e4afc960e099ff730e232" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37715 } } peers { permanent_uuid: "c72c1d866ac34aa3bbcae200c28f8eb3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36695 } } peers { permanent_uuid: "757e3e76bff144399dc79b6855581575" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43191 } }
I20260928 05:17:13.872655 2024 leader_election.cc:288] T e5935a6a6ab44823995e601ea733c89d P 757e3e76bff144399dc79b6855581575 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c6a93ea324e4afc960e099ff730e232 (127.0.97.1:37715), c72c1d866ac34aa3bbcae200c28f8eb3 (127.0.97.2:36695)
I20260928 05:17:13.873207 1673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5935a6a6ab44823995e601ea733c89d" candidate_uuid: "757e3e76bff144399dc79b6855581575" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c6a93ea324e4afc960e099ff730e232" is_pre_election: true
I20260928 05:17:13.873291 1808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5935a6a6ab44823995e601ea733c89d" candidate_uuid: "757e3e76bff144399dc79b6855581575" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c72c1d866ac34aa3bbcae200c28f8eb3" is_pre_election: true
I20260928 05:17:13.873433 1673 raft_consensus.cc:2454] T e5935a6a6ab44823995e601ea733c89d P 9c6a93ea324e4afc960e099ff730e232 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 757e3e76bff144399dc79b6855581575 in term 0.
I20260928 05:17:13.873559 1808 raft_consensus.cc:2454] T e5935a6a6ab44823995e601ea733c89d P c72c1d866ac34aa3bbcae200c28f8eb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 757e3e76bff144399dc79b6855581575 in term 0.
I20260928 05:17:13.873834 1879 leader_election.cc:302] T e5935a6a6ab44823995e601ea733c89d P 757e3e76bff144399dc79b6855581575 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 757e3e76bff144399dc79b6855581575, 9c6a93ea324e4afc960e099ff730e232; no voters:
I20260928 05:17:13.874082 2024 raft_consensus.cc:2781] T e5935a6a6ab44823995e601ea733c89d P 757e3e76bff144399dc79b6855581575 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 05:17:13.874177 2024 raft_consensus.cc:482] T e5935a6a6ab44823995e601ea733c89d P 757e3e76bff144399dc79b6855581575 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 05:17:13.874279 2024 raft_consensus.cc:3037] T e5935a6a6ab44823995e601ea733c89d P 757e3e76bff144399dc79b6855581575 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:17:13.876259 2024 raft_consensus.cc:504] T e5935a6a6ab44823995e601ea733c89d P 757e3e76bff144399dc79b6855581575 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c6a93ea324e4afc960e099ff730e232" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37715 } } peers { permanent_uuid: "c72c1d866ac34aa3bbcae200c28f8eb3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36695 } } peers { permanent_uuid: "757e3e76bff144399dc79b6855581575" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43191 } }
I20260928 05:17:13.876749 2024 leader_election.cc:288] T e5935a6a6ab44823995e601ea733c89d P 757e3e76bff144399dc79b6855581575 [CANDIDATE]: Term 1 election: Requested vote from peers 9c6a93ea324e4afc960e099ff730e232 (127.0.97.1:37715), c72c1d866ac34aa3bbcae200c28f8eb3 (127.0.97.2:36695)
I20260928 05:17:13.877123 1673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5935a6a6ab44823995e601ea733c89d" candidate_uuid: "757e3e76bff144399dc79b6855581575" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c6a93ea324e4afc960e099ff730e232"
I20260928 05:17:13.877243 1808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5935a6a6ab44823995e601ea733c89d" candidate_uuid: "757e3e76bff144399dc79b6855581575" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c72c1d866ac34aa3bbcae200c28f8eb3"
I20260928 05:17:13.877310 1673 raft_consensus.cc:3037] T e5935a6a6ab44823995e601ea733c89d P 9c6a93ea324e4afc960e099ff730e232 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:17:13.877445 1808 raft_consensus.cc:3037] T e5935a6a6ab44823995e601ea733c89d P c72c1d866ac34aa3bbcae200c28f8eb3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:17:13.879340 1673 raft_consensus.cc:2454] T e5935a6a6ab44823995e601ea733c89d P 9c6a93ea324e4afc960e099ff730e232 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 757e3e76bff144399dc79b6855581575 in term 1.
I20260928 05:17:13.879335 1808 raft_consensus.cc:2454] T e5935a6a6ab44823995e601ea733c89d P c72c1d866ac34aa3bbcae200c28f8eb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 757e3e76bff144399dc79b6855581575 in term 1.
I20260928 05:17:13.879648 1879 leader_election.cc:302] T e5935a6a6ab44823995e601ea733c89d P 757e3e76bff144399dc79b6855581575 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 757e3e76bff144399dc79b6855581575, c72c1d866ac34aa3bbcae200c28f8eb3; no voters:
I20260928 05:17:13.879874 2024 raft_consensus.cc:2781] T e5935a6a6ab44823995e601ea733c89d P 757e3e76bff144399dc79b6855581575 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:17:13.880038 2024 raft_consensus.cc:686] T e5935a6a6ab44823995e601ea733c89d P 757e3e76bff144399dc79b6855581575 [term 1 LEADER]: Becoming Leader. State: Replica: 757e3e76bff144399dc79b6855581575, State: Running, Role: LEADER
I20260928 05:17:13.880267 2024 consensus_queue.cc:237] T e5935a6a6ab44823995e601ea733c89d P 757e3e76bff144399dc79b6855581575 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c6a93ea324e4afc960e099ff730e232" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37715 } } peers { permanent_uuid: "c72c1d866ac34aa3bbcae200c28f8eb3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36695 } } peers { permanent_uuid: "757e3e76bff144399dc79b6855581575" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43191 } }
I20260928 05:17:13.881628 2047 tablet_replica.cc:439] T e5935a6a6ab44823995e601ea733c89d P 757e3e76bff144399dc79b6855581575: TxnStatusTablet state changed. Reason: New leader 757e3e76bff144399dc79b6855581575. Latest consensus state: current_term: 1 leader_uuid: "757e3e76bff144399dc79b6855581575" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c6a93ea324e4afc960e099ff730e232" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37715 } } peers { permanent_uuid: "c72c1d866ac34aa3bbcae200c28f8eb3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36695 } } peers { permanent_uuid: "757e3e76bff144399dc79b6855581575" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43191 } } }
I20260928 05:17:13.881834 2047 tablet_replica.cc:442] T e5935a6a6ab44823995e601ea733c89d P 757e3e76bff144399dc79b6855581575: This TxnStatusTablet replica's current role is: LEADER
I20260928 05:17:13.882411 2060 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 05:17:13.882591 2060 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 05:17:13.882886 1509 catalog_manager.cc:5495] T e5935a6a6ab44823995e601ea733c89d P 757e3e76bff144399dc79b6855581575 reported cstate change: term changed from 0 to 1, leader changed from <none> to 757e3e76bff144399dc79b6855581575 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "757e3e76bff144399dc79b6855581575" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c6a93ea324e4afc960e099ff730e232" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37715 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c72c1d866ac34aa3bbcae200c28f8eb3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "757e3e76bff144399dc79b6855581575" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43191 } health_report { overall_health: HEALTHY } } }
I20260928 05:17:13.914392 2055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:13.908800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43191 (local address 127.0.0.1:54114)
0928 05:17:13.909005 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:13.909025 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:13.909126 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:13.909886 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:13.909891 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:13.909903 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 05:17:13.910094 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:13.910104 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:13.911155 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:13.911160 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:13.912133 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:13.912143 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:13.912803 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:13.912810 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:13.912986 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:13.913036 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 05:17:13.913871 (+ 835us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 05:17:13.913891 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 05:17:13.913961 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 05:17:13.914099 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260928 05:17:13.914460 2063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:13.908938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54114 (local address 127.0.97.3:43191)
0928 05:17:13.909554 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:13.909558 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:13.909575 (+ 17us) server_negotiation.cc:407] Connection header received
0928 05:17:13.909642 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:13.909645 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:13.909710 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:13.909855 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:17:13.910236 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:13.911042 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:13.912259 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:13.912670 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:13.912787 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:13.913229 (+ 442us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:17:13.913773 (+ 544us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:17:13.913899 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:13.914020 (+ 121us) server_negotiation.cc:299] Negotiation successful
0928 05:17:13.914096 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":333,"threads_started":1}
I20260928 05:17:13.918339 1673 raft_consensus.cc:1261] T e5935a6a6ab44823995e601ea733c89d P 9c6a93ea324e4afc960e099ff730e232 [term 1 FOLLOWER]: Refusing update from remote peer 757e3e76bff144399dc79b6855581575: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:17:13.918656 1808 raft_consensus.cc:1261] T e5935a6a6ab44823995e601ea733c89d P c72c1d866ac34aa3bbcae200c28f8eb3 [term 1 FOLLOWER]: Refusing update from remote peer 757e3e76bff144399dc79b6855581575: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:17:13.918989 2047 consensus_queue.cc:1035] T e5935a6a6ab44823995e601ea733c89d P 757e3e76bff144399dc79b6855581575 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c6a93ea324e4afc960e099ff730e232" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:17:13.919358 2024 consensus_queue.cc:1035] T e5935a6a6ab44823995e601ea733c89d P 757e3e76bff144399dc79b6855581575 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c72c1d866ac34aa3bbcae200c28f8eb3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:17:13.922118 2027 tablet_replica.cc:439] T e5935a6a6ab44823995e601ea733c89d P c72c1d866ac34aa3bbcae200c28f8eb3: TxnStatusTablet state changed. Reason: New leader 757e3e76bff144399dc79b6855581575. Latest consensus state: current_term: 1 leader_uuid: "757e3e76bff144399dc79b6855581575" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c6a93ea324e4afc960e099ff730e232" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37715 } } peers { permanent_uuid: "c72c1d866ac34aa3bbcae200c28f8eb3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36695 } } peers { permanent_uuid: "757e3e76bff144399dc79b6855581575" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43191 } } }
I20260928 05:17:13.922385 2027 tablet_replica.cc:442] T e5935a6a6ab44823995e601ea733c89d P c72c1d866ac34aa3bbcae200c28f8eb3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 05:17:13.924934 2047 tablet_replica.cc:439] T e5935a6a6ab44823995e601ea733c89d P 757e3e76bff144399dc79b6855581575: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "757e3e76bff144399dc79b6855581575" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c6a93ea324e4afc960e099ff730e232" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37715 } } peers { permanent_uuid: "c72c1d866ac34aa3bbcae200c28f8eb3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36695 } } peers { permanent_uuid: "757e3e76bff144399dc79b6855581575" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43191 } } }
I20260928 05:17:13.925350 2047 tablet_replica.cc:442] T e5935a6a6ab44823995e601ea733c89d P 757e3e76bff144399dc79b6855581575: This TxnStatusTablet replica's current role is: LEADER
I20260928 05:17:13.926230 2032 mvcc.cc:205] Tried to move back new op lower bound from 7334185508522934272 to 7334185508375339008. Current Snapshot: MvccSnapshot[applied={T|T < 7334185508522934272 or (T in {7334185508522934272})}]
I20260928 05:17:13.928825 2030 mvcc.cc:205] Tried to move back new op lower bound from 7334185508522934272 to 7334185508375339008. Current Snapshot: MvccSnapshot[applied={T|T < 7334185508522934272}]
I20260928 05:17:13.929212 2027 tablet_replica.cc:439] T e5935a6a6ab44823995e601ea733c89d P c72c1d866ac34aa3bbcae200c28f8eb3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "757e3e76bff144399dc79b6855581575" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c6a93ea324e4afc960e099ff730e232" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37715 } } peers { permanent_uuid: "c72c1d866ac34aa3bbcae200c28f8eb3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36695 } } peers { permanent_uuid: "757e3e76bff144399dc79b6855581575" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43191 } } }
I20260928 05:17:13.929515 2027 tablet_replica.cc:442] T e5935a6a6ab44823995e601ea733c89d P c72c1d866ac34aa3bbcae200c28f8eb3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 05:17:13.933679 2023 tablet_replica.cc:439] T e5935a6a6ab44823995e601ea733c89d P 9c6a93ea324e4afc960e099ff730e232: TxnStatusTablet state changed. Reason: New leader 757e3e76bff144399dc79b6855581575. Latest consensus state: current_term: 1 leader_uuid: "757e3e76bff144399dc79b6855581575" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c6a93ea324e4afc960e099ff730e232" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37715 } } peers { permanent_uuid: "c72c1d866ac34aa3bbcae200c28f8eb3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36695 } } peers { permanent_uuid: "757e3e76bff144399dc79b6855581575" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43191 } } }
I20260928 05:17:13.933990 2023 tablet_replica.cc:442] T e5935a6a6ab44823995e601ea733c89d P 9c6a93ea324e4afc960e099ff730e232: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 05:17:13.934998 2024 tablet_replica.cc:439] T e5935a6a6ab44823995e601ea733c89d P 757e3e76bff144399dc79b6855581575: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "757e3e76bff144399dc79b6855581575" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c6a93ea324e4afc960e099ff730e232" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37715 } } peers { permanent_uuid: "c72c1d866ac34aa3bbcae200c28f8eb3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36695 } } peers { permanent_uuid: "757e3e76bff144399dc79b6855581575" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43191 } } }
I20260928 05:17:13.935314 2024 tablet_replica.cc:442] T e5935a6a6ab44823995e601ea733c89d P 757e3e76bff144399dc79b6855581575: This TxnStatusTablet replica's current role is: LEADER
I20260928 05:17:13.938663 2023 tablet_replica.cc:439] T e5935a6a6ab44823995e601ea733c89d P 9c6a93ea324e4afc960e099ff730e232: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "757e3e76bff144399dc79b6855581575" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c6a93ea324e4afc960e099ff730e232" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37715 } } peers { permanent_uuid: "c72c1d866ac34aa3bbcae200c28f8eb3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36695 } } peers { permanent_uuid: "757e3e76bff144399dc79b6855581575" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43191 } } }
I20260928 05:17:13.938868 2023 tablet_replica.cc:442] T e5935a6a6ab44823995e601ea733c89d P 9c6a93ea324e4afc960e099ff730e232: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 05:17:13.941214 2063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:13.932656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54130 (local address 127.0.97.3:43191)
0928 05:17:13.932813 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:13.932819 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:13.934983 (+ 2164us) server_negotiation.cc:407] Connection header received
0928 05:17:13.935251 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:13.935257 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:13.935335 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:13.935395 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:17:13.936084 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:13.937162 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:13.939424 (+ 2262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:13.939947 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:13.940043 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:13.940579 (+ 536us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:17:13.940673 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:17:13.940752 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:13.940907 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 05:17:13.940976 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 05:17:13.956773 2063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:13.950898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36481 (local address 127.0.97.3:43191)
0928 05:17:13.951065 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:13.951071 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:13.951625 (+ 554us) server_negotiation.cc:407] Connection header received
0928 05:17:13.951780 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:13.951785 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:13.951876 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:13.951969 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:17:13.952395 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:13.953529 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:13.954921 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:13.955601 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:13.955716 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:13.956092 (+ 376us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:17:13.956203 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:17:13.956281 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:13.956386 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 05:17:13.956453 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 05:17:13.956882 2070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:13.950915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43191 (local address 127.0.97.3:36481)
0928 05:17:13.951538 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:13.951552 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:13.951657 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:13.951978 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:13.951981 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:13.951989 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 05:17:13.952262 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:13.952272 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:13.953641 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:13.953644 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:13.954804 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:13.954815 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:13.955765 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:13.955770 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:13.955883 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:13.955936 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 05:17:13.956295 (+ 359us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 05:17:13.956302 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 05:17:13.956353 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 05:17:13.956475 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":355,"threads_started":1}
I20260928 05:17:13.964956 1808 raft_consensus.cc:1261] T e9efd5f6a9f44a0b929a36705f29f50b P c72c1d866ac34aa3bbcae200c28f8eb3 [term 1 FOLLOWER]: Refusing update from remote peer 757e3e76bff144399dc79b6855581575: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:17:13.964956 1673 raft_consensus.cc:1261] T e9efd5f6a9f44a0b929a36705f29f50b P 9c6a93ea324e4afc960e099ff730e232 [term 1 FOLLOWER]: Refusing update from remote peer 757e3e76bff144399dc79b6855581575: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:17:13.965420 2047 consensus_queue.cc:1035] T e9efd5f6a9f44a0b929a36705f29f50b P 757e3e76bff144399dc79b6855581575 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c72c1d866ac34aa3bbcae200c28f8eb3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:17:13.965667 2024 consensus_queue.cc:1035] T e9efd5f6a9f44a0b929a36705f29f50b P 757e3e76bff144399dc79b6855581575 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c6a93ea324e4afc960e099ff730e232" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:17:14.028016 1510 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55514:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:17:14.028277 1510 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55514:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:17:14.033442 1510 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d09101969d894158a93c73549cbd740f: Sending DeleteTablet for 3 replicas of tablet e9efd5f6a9f44a0b929a36705f29f50b
I20260928 05:17:14.034260 1787 tablet_service.cc:1514] Processing DeleteTablet for tablet e9efd5f6a9f44a0b929a36705f29f50b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:17:14 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53492
I20260928 05:17:14.034628 1648 tablet_service.cc:1514] Processing DeleteTablet for tablet e9efd5f6a9f44a0b929a36705f29f50b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:17:14 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41062
I20260928 05:17:14.035130 1932 tablet_service.cc:1514] Processing DeleteTablet for tablet e9efd5f6a9f44a0b929a36705f29f50b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:17:14 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54108
I20260928 05:17:14.044559 2076 tablet_replica.cc:330] T e9efd5f6a9f44a0b929a36705f29f50b P 9c6a93ea324e4afc960e099ff730e232: stopping tablet replica
I20260928 05:17:14.044787 2076 raft_consensus.cc:2229] T e9efd5f6a9f44a0b929a36705f29f50b P 9c6a93ea324e4afc960e099ff730e232 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:17:14.045053 2076 raft_consensus.cc:2258] T e9efd5f6a9f44a0b929a36705f29f50b P 9c6a93ea324e4afc960e099ff730e232 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:17:14.047359 2076 ts_tablet_manager.cc:1905] T e9efd5f6a9f44a0b929a36705f29f50b P 9c6a93ea324e4afc960e099ff730e232: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:17:14.049333 2077 tablet_replica.cc:330] T e9efd5f6a9f44a0b929a36705f29f50b P 757e3e76bff144399dc79b6855581575: stopping tablet replica
I20260928 05:17:14.049630 2077 raft_consensus.cc:2229] T e9efd5f6a9f44a0b929a36705f29f50b P 757e3e76bff144399dc79b6855581575 [term 1 LEADER]: Raft consensus shutting down.
I20260928 05:17:14.050230 2077 raft_consensus.cc:2258] T e9efd5f6a9f44a0b929a36705f29f50b P 757e3e76bff144399dc79b6855581575 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:17:14.051273 2077 ts_tablet_manager.cc:1905] T e9efd5f6a9f44a0b929a36705f29f50b P 757e3e76bff144399dc79b6855581575: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:17:14.053908 2075 tablet_replica.cc:330] T e9efd5f6a9f44a0b929a36705f29f50b P c72c1d866ac34aa3bbcae200c28f8eb3: stopping tablet replica
I20260928 05:17:14.054826 2075 raft_consensus.cc:2229] T e9efd5f6a9f44a0b929a36705f29f50b P c72c1d866ac34aa3bbcae200c28f8eb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:17:14.054955 2076 ts_tablet_manager.cc:1918] T e9efd5f6a9f44a0b929a36705f29f50b P 9c6a93ea324e4afc960e099ff730e232: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 05:17:14.055071 2076 log.cc:1196] T e9efd5f6a9f44a0b929a36705f29f50b P 9c6a93ea324e4afc960e099ff730e232: Deleting WAL directory at /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/minicluster-data/ts-0/wal/wals/e9efd5f6a9f44a0b929a36705f29f50b
I20260928 05:17:14.055222 2075 raft_consensus.cc:2258] T e9efd5f6a9f44a0b929a36705f29f50b P c72c1d866ac34aa3bbcae200c28f8eb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:17:14.055490 2076 ts_tablet_manager.cc:1939] T e9efd5f6a9f44a0b929a36705f29f50b P 9c6a93ea324e4afc960e099ff730e232: Deleting consensus metadata
I20260928 05:17:14.056387 2075 ts_tablet_manager.cc:1905] T e9efd5f6a9f44a0b929a36705f29f50b P c72c1d866ac34aa3bbcae200c28f8eb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:17:14.056558 1496 catalog_manager.cc:4841] TS 9c6a93ea324e4afc960e099ff730e232 (127.0.97.1:37715): tablet e9efd5f6a9f44a0b929a36705f29f50b (table test-table [id=2c6d1403a7874789a4be1eaa0cfd44b7]) successfully deleted
I20260928 05:17:14.059746 2077 ts_tablet_manager.cc:1918] T e9efd5f6a9f44a0b929a36705f29f50b P 757e3e76bff144399dc79b6855581575: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 05:17:14.059857 2077 log.cc:1196] T e9efd5f6a9f44a0b929a36705f29f50b P 757e3e76bff144399dc79b6855581575: Deleting WAL directory at /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/minicluster-data/ts-2/wal/wals/e9efd5f6a9f44a0b929a36705f29f50b
I20260928 05:17:14.060218 2077 ts_tablet_manager.cc:1939] T e9efd5f6a9f44a0b929a36705f29f50b P 757e3e76bff144399dc79b6855581575: Deleting consensus metadata
I20260928 05:17:14.061134 1495 catalog_manager.cc:4841] TS 757e3e76bff144399dc79b6855581575 (127.0.97.3:43191): tablet e9efd5f6a9f44a0b929a36705f29f50b (table test-table [id=2c6d1403a7874789a4be1eaa0cfd44b7]) successfully deleted
Sep 28 05:17:14 dist-test-slave-whsn krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572633, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260928 05:17:14.065989 2075 ts_tablet_manager.cc:1918] T e9efd5f6a9f44a0b929a36705f29f50b P c72c1d866ac34aa3bbcae200c28f8eb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 05:17:14.066077 2075 log.cc:1196] T e9efd5f6a9f44a0b929a36705f29f50b P c72c1d866ac34aa3bbcae200c28f8eb3: Deleting WAL directory at /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572622576160-388-0/minicluster-data/ts-1/wal/wals/e9efd5f6a9f44a0b929a36705f29f50b
I20260928 05:17:14.066346 2075 ts_tablet_manager.cc:1939] T e9efd5f6a9f44a0b929a36705f29f50b P c72c1d866ac34aa3bbcae200c28f8eb3: Deleting consensus metadata
I20260928 05:17:14.067113 1496 catalog_manager.cc:4841] TS c72c1d866ac34aa3bbcae200c28f8eb3 (127.0.97.2:36695): tablet e9efd5f6a9f44a0b929a36705f29f50b (table test-table [id=2c6d1403a7874789a4be1eaa0cfd44b7]) successfully deleted
I20260928 05:17:14.071942 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:14.046668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41064 (local address 127.0.97.1:37715)
0928 05:17:14.046828 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:14.046834 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:14.053408 (+ 6574us) server_negotiation.cc:407] Connection header received
0928 05:17:14.054020 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:14.054026 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:14.054111 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:14.054204 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:14.056536 (+ 2332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:14.057662 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:14.059534 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:14.060139 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:14.060227 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:14.065346 (+ 5119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:14.065362 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:14.065382 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:14.065450 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:14.069516 (+ 4066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:14.070096 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:14.070104 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:14.070107 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:14.070261 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:14.070575 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:14.070580 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:14.070583 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:14.070971 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:14.071238 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:14.071383 (+ 145us) server_negotiation.cc:299] Negotiation successful
0928 05:17:14.071592 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260928 05:17:14.072722 1713 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41064
I20260928 05:17:14.088982 2054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:14.075491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55528 (local address 127.0.97.62:39279)
0928 05:17:14.075736 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:14.075740 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:14.075864 (+ 124us) server_negotiation.cc:407] Connection header received
0928 05:17:14.076017 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:14.076020 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:14.076077 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:14.076159 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:14.077680 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:14.078455 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:14.080134 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:14.080578 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:14.080667 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:14.083654 (+ 2987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:14.083659 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:14.083662 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:14.083696 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:14.086556 (+ 2860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:14.087238 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:14.087242 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:14.087244 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:14.087355 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:14.087710 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:14.087713 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:14.087715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:14.087994 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:14.088176 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:14.088522 (+ 346us) server_negotiation.cc:299] Negotiation successful
0928 05:17:14.088690 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133}
W20260928 05:17:14.089511 1510 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55528
I20260928 05:17:14.091468 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 1582
I20260928 05:17:14.104496 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:14.092538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41080 (local address 127.0.97.1:37715)
0928 05:17:14.092722 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:14.092729 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:14.093009 (+ 280us) server_negotiation.cc:407] Connection header received
0928 05:17:14.093140 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:14.093146 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:14.093232 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:14.093355 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:14.094294 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:14.095402 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:14.096476 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:14.097119 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:14.097237 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:14.099067 (+ 1830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:14.099073 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:14.099079 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:14.099113 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:14.102357 (+ 3244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:14.102894 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:14.102899 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:14.102903 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:14.103035 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:14.103293 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:14.103297 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:14.103301 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:14.103580 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:14.103813 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:14.103954 (+ 141us) server_negotiation.cc:299] Negotiation successful
0928 05:17:14.104146 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260928 05:17:14.105126 1713 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41080
W20260928 05:17:14.105587 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 1582: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 05:17:14.105741 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 1582
I20260928 05:17:14.119719 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 1722
Sep 28 05:17:14 dist-test-slave-whsn krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572633, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260928 05:17:14.133317 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:14.120442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53508 (local address 127.0.97.2:36695)
0928 05:17:14.120582 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:14.120586 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:14.120698 (+ 112us) server_negotiation.cc:407] Connection header received
0928 05:17:14.120911 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:14.120916 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:14.120985 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:14.121063 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:14.122149 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:14.122938 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:14.124073 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:14.124470 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:14.124573 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:14.128134 (+ 3561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:14.128146 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:14.128159 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:14.128193 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:14.130762 (+ 2569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:14.131358 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:14.131367 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:14.131369 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:14.131488 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:14.131821 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:14.131826 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:14.131829 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:14.132164 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:14.132434 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:14.132696 (+ 262us) server_negotiation.cc:299] Negotiation successful
0928 05:17:14.133034 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
W20260928 05:17:14.133893 1853 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53508
W20260928 05:17:14.134472 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 1722: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 05:17:14.134588 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 1722
I20260928 05:17:14.159056 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 1862
Sep 28 05:17:14 dist-test-slave-whsn krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572633, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260928 05:17:14.178915 2063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:14.159766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54144 (local address 127.0.97.3:43191)
0928 05:17:14.159956 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:14.159962 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:14.161214 (+ 1252us) server_negotiation.cc:407] Connection header received
0928 05:17:14.161397 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:14.161402 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:14.161486 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:14.161575 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:14.163035 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:14.164210 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:14.166014 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:14.166713 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:14.166829 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:14.171884 (+ 5055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:14.171896 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:14.171914 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:14.171953 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:14.175670 (+ 3717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:14.176444 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:14.176453 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:14.176457 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:14.176627 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:14.177071 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:14.177076 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:14.177080 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:14.177571 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:14.177926 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:14.178133 (+ 207us) server_negotiation.cc:299] Negotiation successful
0928 05:17:14.178377 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260928 05:17:14.179623 1992 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54144
W20260928 05:17:14.180336 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 1862: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 05:17:14.180497 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 1862
I20260928 05:17:14.195621 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 1479
I20260928 05:17:14.196560 1551 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:17:14.305204 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 1479
2026-09-28T05:17:14Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4312 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:17:14 dist-test-slave-whsn krb5kdc[2086](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:17:14 dist-test-slave-whsn krb5kdc[2086](info): set up 2 sockets
Sep 28 05:17:14 dist-test-slave-whsn krb5kdc[2086](info): commencing operation
krb5kdc: starting...
W20260928 05:17:15.288565 388 mini_kdc.cc:121] Time spent starting KDC: real 0.946s user 0.001s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/krb5kdc/test-user.keytab.
Sep 28 05:17:15 dist-test-slave-whsn krb5kdc[2086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572635, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:17:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:17:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 05:17:15.440196 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35227
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35227
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:17:15.731004 2099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:17:15.731276 2099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:17:15.731400 2099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:17:15.738205 2099 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:17:15.738302 2099 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 05:17:15.738365 2099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:17:15.738448 2099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:17:15.738508 2099 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:17:15.738585 2099 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:17:15.747749 2099 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35227
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:17:15.748816 2099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:17:15.750061 2099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:17:15.760666 2105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:15.760644 2107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:15.760819 2104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:17:15.761513 2099 server_base.cc:1034] running on GCE node
I20260928 05:17:15.762182 2099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:17:15.763337 2099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:17:15.764539 2099 hybrid_clock.cc:648] HybridClock initialized: now 1790572635764480 us; error 38 us; skew 500 ppm
I20260928 05:17:15.764825 2099 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:17:15 dist-test-slave-whsn krb5kdc[2086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572635, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:17:15.771819 2099 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 05:17:15.773093 2099 webserver.cc:460] Webserver started at http://127.0.97.62:34063/ using document root <none> and password file <none>
I20260928 05:17:15.773693 2099 fs_manager.cc:359] Metadata directory not provided
I20260928 05:17:15.773777 2099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:17:15.774056 2099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:17:15.776687 2099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/minicluster-data/master-0/data/instance:
uuid: "aad10d5c9dbc434e9742c4102ad69acd"
format_stamp: "Formatted at 2026-09-28 05:17:15 on dist-test-slave-whsn"
server_key: "3495eced25aa1be541e41fb8b8427783"
server_key_iv: "82d4aa20789ff179a6f93134d64bd064"
server_key_version: "encryptionkey@0"
I20260928 05:17:15.777419 2099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/minicluster-data/master-0/wal/instance:
uuid: "aad10d5c9dbc434e9742c4102ad69acd"
format_stamp: "Formatted at 2026-09-28 05:17:15 on dist-test-slave-whsn"
server_key: "3495eced25aa1be541e41fb8b8427783"
server_key_iv: "82d4aa20789ff179a6f93134d64bd064"
server_key_version: "encryptionkey@0"
I20260928 05:17:15.781831 2099 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 05:17:15.784795 2114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:15.785957 2099 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 05:17:15.786092 2099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/minicluster-data/master-0/wal
uuid: "aad10d5c9dbc434e9742c4102ad69acd"
format_stamp: "Formatted at 2026-09-28 05:17:15 on dist-test-slave-whsn"
server_key: "3495eced25aa1be541e41fb8b8427783"
server_key_iv: "82d4aa20789ff179a6f93134d64bd064"
server_key_version: "encryptionkey@0"
I20260928 05:17:15.786243 2099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:17:15.798388 2099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:17:15.802124 2099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:17:15.802318 2099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:17:15.820235 2099 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35227
I20260928 05:17:15.820254 2175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35227 every 8 connection(s)
I20260928 05:17:15.821457 2099 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:17:15.824817 2176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:17:15.831781 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 2099
I20260928 05:17:15.831969 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/minicluster-data/master-0/wal/instance
I20260928 05:17:15.832535 388 external_mini_cluster.cc:1428] Setting key 1ebfc6c70f8031cf6bce359292685da9
I20260928 05:17:15.834127 2176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aad10d5c9dbc434e9742c4102ad69acd: Bootstrap starting.
I20260928 05:17:15.836968 2176 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aad10d5c9dbc434e9742c4102ad69acd: Neither blocks nor log segments found. Creating new log.
I20260928 05:17:15.838068 2176 log.cc:824] T 00000000000000000000000000000000 P aad10d5c9dbc434e9742c4102ad69acd: Log is configured to *not* fsync() on all Append() calls
I20260928 05:17:15.840528 2176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aad10d5c9dbc434e9742c4102ad69acd: No bootstrap required, opened a new log
Sep 28 05:17:15 dist-test-slave-whsn krb5kdc[2086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572635, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:17:15.849563 2176 raft_consensus.cc:348] T 00000000000000000000000000000000 P aad10d5c9dbc434e9742c4102ad69acd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aad10d5c9dbc434e9742c4102ad69acd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35227 } }
I20260928 05:17:15.849979 2176 raft_consensus.cc:374] T 00000000000000000000000000000000 P aad10d5c9dbc434e9742c4102ad69acd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:17:15.850082 2176 raft_consensus.cc:729] T 00000000000000000000000000000000 P aad10d5c9dbc434e9742c4102ad69acd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad10d5c9dbc434e9742c4102ad69acd, State: Initialized, Role: FOLLOWER
I20260928 05:17:15.850679 2176 consensus_queue.cc:260] T 00000000000000000000000000000000 P aad10d5c9dbc434e9742c4102ad69acd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aad10d5c9dbc434e9742c4102ad69acd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35227 } }
I20260928 05:17:15.850876 2176 raft_consensus.cc:388] T 00000000000000000000000000000000 P aad10d5c9dbc434e9742c4102ad69acd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:17:15.851003 2176 raft_consensus.cc:482] T 00000000000000000000000000000000 P aad10d5c9dbc434e9742c4102ad69acd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:17:15.851161 2176 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aad10d5c9dbc434e9742c4102ad69acd [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:17:15.853327 2176 raft_consensus.cc:504] T 00000000000000000000000000000000 P aad10d5c9dbc434e9742c4102ad69acd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aad10d5c9dbc434e9742c4102ad69acd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35227 } }
I20260928 05:17:15.853785 2176 leader_election.cc:302] T 00000000000000000000000000000000 P aad10d5c9dbc434e9742c4102ad69acd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aad10d5c9dbc434e9742c4102ad69acd; no voters:
I20260928 05:17:15.854025 2179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:15.834250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59590 (local address 127.0.97.62:35227)
0928 05:17:15.834987 (+ 737us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:15.834997 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:15.835034 (+ 37us) server_negotiation.cc:407] Connection header received
0928 05:17:15.836651 (+ 1617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:15.836672 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:15.837053 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:15.837374 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:15.838997 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:15.840247 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:15.841888 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:15.842590 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:15.842714 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:15.846794 (+ 4080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:15.846804 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:15.846821 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:15.846864 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:15.850354 (+ 3490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:15.850937 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:15.850945 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:15.850952 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:15.851153 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:15.851472 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:15.851478 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:15.851481 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:15.851990 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:15.852271 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:15.852648 (+ 377us) server_negotiation.cc:299] Negotiation successful
0928 05:17:15.853041 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":345,"threads_started":1}
I20260928 05:17:15.854329 2176 leader_election.cc:288] T 00000000000000000000000000000000 P aad10d5c9dbc434e9742c4102ad69acd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:17:15.854434 2181 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aad10d5c9dbc434e9742c4102ad69acd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:17:15.855602 2181 raft_consensus.cc:686] T 00000000000000000000000000000000 P aad10d5c9dbc434e9742c4102ad69acd [term 1 LEADER]: Becoming Leader. State: Replica: aad10d5c9dbc434e9742c4102ad69acd, State: Running, Role: LEADER
I20260928 05:17:15.856117 2181 consensus_queue.cc:237] T 00000000000000000000000000000000 P aad10d5c9dbc434e9742c4102ad69acd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aad10d5c9dbc434e9742c4102ad69acd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35227 } }
I20260928 05:17:15.856302 2176 sys_catalog.cc:564] T 00000000000000000000000000000000 P aad10d5c9dbc434e9742c4102ad69acd [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:17:15.858703 2183 sys_catalog.cc:455] T 00000000000000000000000000000000 P aad10d5c9dbc434e9742c4102ad69acd [sys.catalog]: SysCatalogTable state changed. Reason: New leader aad10d5c9dbc434e9742c4102ad69acd. Latest consensus state: current_term: 1 leader_uuid: "aad10d5c9dbc434e9742c4102ad69acd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aad10d5c9dbc434e9742c4102ad69acd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35227 } } }
I20260928 05:17:15.858771 2182 sys_catalog.cc:455] T 00000000000000000000000000000000 P aad10d5c9dbc434e9742c4102ad69acd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aad10d5c9dbc434e9742c4102ad69acd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aad10d5c9dbc434e9742c4102ad69acd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35227 } } }
I20260928 05:17:15.859030 2183 sys_catalog.cc:458] T 00000000000000000000000000000000 P aad10d5c9dbc434e9742c4102ad69acd [sys.catalog]: This master's current role is: LEADER
I20260928 05:17:15.859071 2182 sys_catalog.cc:458] T 00000000000000000000000000000000 P aad10d5c9dbc434e9742c4102ad69acd [sys.catalog]: This master's current role is: LEADER
I20260928 05:17:15.860702 2186 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:17:15.865855 2186 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:17:15.873605 2186 catalog_manager.cc:1347] Generated new cluster ID: 5d1b8add6f3a4b64baa4e12747ea014f
I20260928 05:17:15.873692 2186 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:17:15.884821 2186 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:17:15.886555 2186 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:17:15.893558 2186 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aad10d5c9dbc434e9742c4102ad69acd: Generated new TSK 0
I20260928 05:17:15.894352 2186 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260928 05:17:15.940254 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35227
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:17:16.259300 2202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:17:16.259555 2202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:17:16.259697 2202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:17:16.266741 2202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:17:16.266848 2202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:17:16.266965 2202 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 05:17:16.267030 2202 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 05:17:16.267127 2202 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 05:17:16.276307 2202 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:35227
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:17:16.277638 2202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:17:16.278877 2202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:17:16.290868 2207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:16.290830 2210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:16.290900 2208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:17:16.291752 2202 server_base.cc:1034] running on GCE node
I20260928 05:17:16.292281 2202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:17:16.293133 2202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:17:16.294325 2202 hybrid_clock.cc:648] HybridClock initialized: now 1790572636294276 us; error 44 us; skew 500 ppm
I20260928 05:17:16.294616 2202 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:17:16 dist-test-slave-whsn krb5kdc[2086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572636, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:17:16.297463 2202 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260928 05:17:16.298693 2202 webserver.cc:460] Webserver started at http://127.0.97.1:42637/ using document root <none> and password file <none>
I20260928 05:17:16.299264 2202 fs_manager.cc:359] Metadata directory not provided
I20260928 05:17:16.299352 2202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:17:16.299600 2202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:17:16.302289 2202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/minicluster-data/ts-0/data/instance:
uuid: "64aee4950dd845c6b9174d260e230a36"
format_stamp: "Formatted at 2026-09-28 05:17:16 on dist-test-slave-whsn"
server_key: "bf9bdf72eec43c44e05681aebb68e501"
server_key_iv: "ceeea03e4f39fe4cee8f7252e97607b4"
server_key_version: "encryptionkey@0"
I20260928 05:17:16.302963 2202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/minicluster-data/ts-0/wal/instance:
uuid: "64aee4950dd845c6b9174d260e230a36"
format_stamp: "Formatted at 2026-09-28 05:17:16 on dist-test-slave-whsn"
server_key: "bf9bdf72eec43c44e05681aebb68e501"
server_key_iv: "ceeea03e4f39fe4cee8f7252e97607b4"
server_key_version: "encryptionkey@0"
I20260928 05:17:16.307588 2202 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260928 05:17:16.310670 2217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:16.311874 2202 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 05:17:16.311990 2202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/minicluster-data/ts-0/wal
uuid: "64aee4950dd845c6b9174d260e230a36"
format_stamp: "Formatted at 2026-09-28 05:17:16 on dist-test-slave-whsn"
server_key: "bf9bdf72eec43c44e05681aebb68e501"
server_key_iv: "ceeea03e4f39fe4cee8f7252e97607b4"
server_key_version: "encryptionkey@0"
I20260928 05:17:16.312134 2202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:17:16.328169 2202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:17:16.331844 2202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:17:16.332057 2202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:17:16.334371 2202 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:17:16.334455 2202 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:16.334637 2202 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:17:16.334712 2202 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 05:17:16 dist-test-slave-whsn krb5kdc[2086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572636, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:17:16.355031 2179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:16.334761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36171 (local address 127.0.97.62:35227)
0928 05:17:16.334914 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:16.334919 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:16.336339 (+ 1420us) server_negotiation.cc:407] Connection header received
0928 05:17:16.338307 (+ 1968us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:16.338313 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:16.338398 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:16.338503 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:16.340434 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:16.341535 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:16.343792 (+ 2257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:16.344327 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:16.344450 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:16.348706 (+ 4256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:16.348711 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:16.348714 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:16.348745 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:16.351629 (+ 2884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:16.352533 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:16.352538 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:16.352542 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:16.352659 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:16.353238 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:16.353243 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:16.353246 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:16.353586 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:16.353722 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:16.354507 (+ 785us) server_negotiation.cc:299] Negotiation successful
0928 05:17:16.354702 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 05:17:16.356549 2226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:16.335357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35227 (local address 127.0.97.1:36171)
0928 05:17:16.336173 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:16.336218 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:16.338047 (+ 1829us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:16.338731 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:16.338739 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:16.339419 (+ 680us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:17:16.340200 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:16.340223 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:16.341696 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:16.341702 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:16.343660 (+ 1958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:16.343672 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:16.344551 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:16.344558 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:16.344839 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:16.345566 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:17:16.345599 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:17:16.348506 (+ 2907us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:17:16.351780 (+ 3274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:16.351788 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:16.351804 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:16.352365 (+ 561us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:16.352778 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:16.352784 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:16.352788 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:16.353099 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:16.353703 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:17:16.353710 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:17:16.354154 (+ 444us) client_negotiation.cc:769] Sending connection context
0928 05:17:16.354414 (+ 260us) client_negotiation.cc:240] Negotiation successful
0928 05:17:16.354916 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":344,"threads_started":1}
I20260928 05:17:16.370584 2202 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43645
I20260928 05:17:16.370662 2336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43645 every 8 connection(s)
I20260928 05:17:16.371906 2202 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/minicluster-data/ts-0/data/info.pb
I20260928 05:17:16.373855 2337 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35227
I20260928 05:17:16.374056 2337 heartbeater.cc:464] Registering TS with master...
I20260928 05:17:16.374730 2337 heartbeater.cc:511] Master 127.0.97.62:35227 requested a full tablet report, sending...
I20260928 05:17:16.376488 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 2202
I20260928 05:17:16.376595 2131 ts_manager.cc:194] Registered new tserver with Master: 64aee4950dd845c6b9174d260e230a36 (127.0.97.1:43645)
I20260928 05:17:16.376690 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/minicluster-data/ts-0/wal/instance
I20260928 05:17:16.377437 388 external_mini_cluster.cc:1428] Setting key 95b1f558c4ee166eca7cab849142cf2b
I20260928 05:17:16.378135 2131 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:36171
I20260928 05:17:16.381019 2179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:16.367536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36019 (local address 127.0.97.62:35227)
0928 05:17:16.367688 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:16.367694 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:16.368489 (+ 795us) server_negotiation.cc:407] Connection header received
0928 05:17:16.368741 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:16.368746 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:16.368819 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:16.368907 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:16.370050 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:16.371009 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:16.372831 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:16.373418 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:16.373494 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:16.375382 (+ 1888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:16.375388 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:16.375393 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:16.375422 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:16.378458 (+ 3036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:16.379033 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:16.379038 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:16.379042 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:16.379210 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:16.379586 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:16.379591 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:16.379595 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:16.379885 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:16.380022 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:16.380460 (+ 438us) server_negotiation.cc:299] Negotiation successful
0928 05:17:16.380665 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 05:17:16.381073 2329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:16.367629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35227 (local address 127.0.97.1:36019)
0928 05:17:16.368367 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:16.368394 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:16.368592 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:16.369087 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:16.369090 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:16.369507 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:17:16.369880 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:16.369893 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:16.371130 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:16.371135 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:16.372648 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:16.372667 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:16.373799 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:16.373805 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:16.373948 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:16.374514 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:17:16.374529 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:17:16.375232 (+ 703us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:17:16.378570 (+ 3338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:16.378575 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:16.378579 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:16.378906 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:16.379342 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:16.379345 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:16.379347 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:16.379477 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:16.380024 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:17:16.380029 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:17:16.380224 (+ 195us) client_negotiation.cc:769] Sending connection context
0928 05:17:16.380354 (+ 130us) client_negotiation.cc:240] Negotiation successful
0928 05:17:16.380624 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":429,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260928 05:17:16.411702 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35227
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:17:16.708081 2341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:17:16.708339 2341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:17:16.708473 2341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:17:16.715338 2341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:17:16.715446 2341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:17:16.715562 2341 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 05:17:16.715626 2341 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 05:17:16.715726 2341 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 05:17:16.724934 2341 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:35227
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:17:16.726120 2341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:17:16.727312 2341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:17:16.738940 2346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:16.738973 2347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:16.738998 2349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:17:16.739684 2341 server_base.cc:1034] running on GCE node
I20260928 05:17:16.740192 2341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:17:16.741008 2341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:17:16.742182 2341 hybrid_clock.cc:648] HybridClock initialized: now 1790572636742133 us; error 45 us; skew 500 ppm
I20260928 05:17:16.742439 2341 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:17:16 dist-test-slave-whsn krb5kdc[2086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572636, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:17:16.745260 2341 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260928 05:17:16.746461 2341 webserver.cc:460] Webserver started at http://127.0.97.2:36163/ using document root <none> and password file <none>
I20260928 05:17:16.747025 2341 fs_manager.cc:359] Metadata directory not provided
I20260928 05:17:16.747120 2341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:17:16.747344 2341 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:17:16.749914 2341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/minicluster-data/ts-1/data/instance:
uuid: "af01fe7dfaeb464392cfef10732beb0d"
format_stamp: "Formatted at 2026-09-28 05:17:16 on dist-test-slave-whsn"
server_key: "4c7fceb5055b759083fd14a1aa3ef32b"
server_key_iv: "6c13379890bec2086b2acf4cb8876f91"
server_key_version: "encryptionkey@0"
I20260928 05:17:16.750588 2341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/minicluster-data/ts-1/wal/instance:
uuid: "af01fe7dfaeb464392cfef10732beb0d"
format_stamp: "Formatted at 2026-09-28 05:17:16 on dist-test-slave-whsn"
server_key: "4c7fceb5055b759083fd14a1aa3ef32b"
server_key_iv: "6c13379890bec2086b2acf4cb8876f91"
server_key_version: "encryptionkey@0"
I20260928 05:17:16.755113 2341 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260928 05:17:16.758150 2356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:16.759439 2341 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 05:17:16.759557 2341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/minicluster-data/ts-1/wal
uuid: "af01fe7dfaeb464392cfef10732beb0d"
format_stamp: "Formatted at 2026-09-28 05:17:16 on dist-test-slave-whsn"
server_key: "4c7fceb5055b759083fd14a1aa3ef32b"
server_key_iv: "6c13379890bec2086b2acf4cb8876f91"
server_key_version: "encryptionkey@0"
I20260928 05:17:16.759696 2341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:17:16.776890 2341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:17:16.780105 2341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:17:16.780308 2341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:17:16.782510 2341 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:17:16.782593 2341 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:16.782697 2341 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:17:16.782765 2341 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 05:17:16 dist-test-slave-whsn krb5kdc[2086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572636, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:17:16.804786 2179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:16.783036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57885 (local address 127.0.97.62:35227)
0928 05:17:16.783216 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:16.783222 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:16.784455 (+ 1233us) server_negotiation.cc:407] Connection header received
0928 05:17:16.786834 (+ 2379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:16.786840 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:16.786923 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:16.786996 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:16.789267 (+ 2271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:16.790287 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:16.793082 (+ 2795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:16.793567 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:16.793690 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:16.798262 (+ 4572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:16.798267 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:16.798270 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:16.798302 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:16.801414 (+ 3112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:16.802277 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:16.802284 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:16.802288 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:16.802448 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:16.802895 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:16.802899 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:16.802901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:16.803138 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:16.803276 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:16.804198 (+ 922us) server_negotiation.cc:299] Negotiation successful
0928 05:17:16.804439 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 05:17:16.806734 2365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:16.783512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35227 (local address 127.0.97.2:57885)
0928 05:17:16.784289 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:16.784332 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:16.786518 (+ 2186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:16.787506 (+ 988us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:16.787519 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:16.788274 (+ 755us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:17:16.789106 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:16.789135 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:16.790492 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:16.790502 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:16.792908 (+ 2406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:16.792928 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:16.793944 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:16.793952 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:16.794241 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:16.794958 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:17:16.794979 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:17:16.798041 (+ 3062us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:17:16.801566 (+ 3525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:16.801575 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:16.801592 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:16.802134 (+ 542us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:16.802555 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:16.802558 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:16.802561 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:16.802776 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:16.803254 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:17:16.803266 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:17:16.803758 (+ 492us) client_negotiation.cc:769] Sending connection context
0928 05:17:16.804078 (+ 320us) client_negotiation.cc:240] Negotiation successful
0928 05:17:16.804642 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":323,"threads_started":1}
I20260928 05:17:16.818154 2341 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37821
I20260928 05:17:16.818213 2475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37821 every 8 connection(s)
I20260928 05:17:16.819464 2341 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/minicluster-data/ts-1/data/info.pb
I20260928 05:17:16.821386 2476 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35227
I20260928 05:17:16.822023 2476 heartbeater.cc:464] Registering TS with master...
I20260928 05:17:16.822729 2476 heartbeater.cc:511] Master 127.0.97.62:35227 requested a full tablet report, sending...
I20260928 05:17:16.823987 2130 ts_manager.cc:194] Registered new tserver with Master: af01fe7dfaeb464392cfef10732beb0d (127.0.97.2:37821)
I20260928 05:17:16.824887 2130 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:57885
I20260928 05:17:16.827178 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 2341
I20260928 05:17:16.827356 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/minicluster-data/ts-1/wal/instance
I20260928 05:17:16.827854 388 external_mini_cluster.cc:1428] Setting key 6655e49f2f715fbaa9d73e8b8014d901
I20260928 05:17:16.829810 2458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:16.812762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35227 (local address 127.0.97.2:52353)
0928 05:17:16.813472 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:16.813493 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:16.813684 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:16.814053 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:16.814057 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:16.814704 (+ 647us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:17:16.815091 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:16.815106 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:16.816333 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:16.816339 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:16.817927 (+ 1588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:16.817942 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:16.818733 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:16.818739 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:16.818905 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:16.819419 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:17:16.819439 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:17:16.820488 (+ 1049us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:17:16.827217 (+ 6729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:16.827222 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:16.827225 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:16.827698 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:16.828096 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:16.828100 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:16.828104 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:16.828234 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:16.828900 (+ 666us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:17:16.828905 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:17:16.829045 (+ 140us) client_negotiation.cc:769] Sending connection context
0928 05:17:16.829158 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 05:17:16.829416 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":398,"threads_started":1}
I20260928 05:17:16.833293 2179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:16.812619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52353 (local address 127.0.97.62:35227)
0928 05:17:16.812762 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:16.812766 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:16.813570 (+ 804us) server_negotiation.cc:407] Connection header received
0928 05:17:16.813849 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:16.813855 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:16.813921 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:16.814013 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:16.815241 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:16.816201 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:16.818066 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:16.818556 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:16.818652 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:16.820976 (+ 2324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:16.820982 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:16.820986 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:16.821023 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:16.827073 (+ 6050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:16.827825 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:16.827830 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:16.827833 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:16.827974 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:16.828353 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:16.828358 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:16.828361 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:16.828703 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:16.828829 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:16.829263 (+ 434us) server_negotiation.cc:299] Negotiation successful
0928 05:17:16.832968 (+ 3705us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260928 05:17:16.862103 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35227
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:17:17.158836 2480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:17:17.159113 2480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:17:17.159252 2480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:17:17.166256 2480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:17:17.166363 2480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:17:17.166483 2480 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 05:17:17.166558 2480 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 05:17:17.166656 2480 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 05:17:17.175836 2480 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:35227
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:17:17.177058 2480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:17:17.178282 2480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:17:17.189936 2488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:17.190027 2486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:17.190034 2485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:17:17.190883 2480 server_base.cc:1034] running on GCE node
I20260928 05:17:17.191443 2480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:17:17.192315 2480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:17:17.193492 2480 hybrid_clock.cc:648] HybridClock initialized: now 1790572637193451 us; error 42 us; skew 500 ppm
I20260928 05:17:17.193747 2480 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:17:17 dist-test-slave-whsn krb5kdc[2086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572637, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:17:17.196482 2480 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260928 05:17:17.197713 2480 webserver.cc:460] Webserver started at http://127.0.97.3:43439/ using document root <none> and password file <none>
I20260928 05:17:17.198274 2480 fs_manager.cc:359] Metadata directory not provided
I20260928 05:17:17.198362 2480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:17:17.198580 2480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:17:17.201152 2480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/minicluster-data/ts-2/data/instance:
uuid: "3de53bd0c5ee445fbfbcb66500285846"
format_stamp: "Formatted at 2026-09-28 05:17:17 on dist-test-slave-whsn"
server_key: "278191b843f76b9a06b38cc2f6994007"
server_key_iv: "20b7f60b70d687c12248a6bdcb561c63"
server_key_version: "encryptionkey@0"
I20260928 05:17:17.201799 2480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/minicluster-data/ts-2/wal/instance:
uuid: "3de53bd0c5ee445fbfbcb66500285846"
format_stamp: "Formatted at 2026-09-28 05:17:17 on dist-test-slave-whsn"
server_key: "278191b843f76b9a06b38cc2f6994007"
server_key_iv: "20b7f60b70d687c12248a6bdcb561c63"
server_key_version: "encryptionkey@0"
I20260928 05:17:17.206243 2480 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 05:17:17.209182 2495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:17.210337 2480 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 05:17:17.210471 2480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/minicluster-data/ts-2/wal
uuid: "3de53bd0c5ee445fbfbcb66500285846"
format_stamp: "Formatted at 2026-09-28 05:17:17 on dist-test-slave-whsn"
server_key: "278191b843f76b9a06b38cc2f6994007"
server_key_iv: "20b7f60b70d687c12248a6bdcb561c63"
server_key_version: "encryptionkey@0"
I20260928 05:17:17.210623 2480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:17:17.234524 2480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:17:17.237797 2480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:17:17.238031 2480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:17:17.240294 2480 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:17:17.240376 2480 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:17.240515 2480 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:17:17.240640 2480 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 05:17:17 dist-test-slave-whsn krb5kdc[2086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572637, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:17:17.261201 2179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:17.240667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34271 (local address 127.0.97.62:35227)
0928 05:17:17.240843 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:17.240870 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:17.242062 (+ 1192us) server_negotiation.cc:407] Connection header received
0928 05:17:17.243972 (+ 1910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:17.243978 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:17.244060 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:17.244170 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:17.246125 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:17.247071 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:17.249391 (+ 2320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:17.250047 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:17.250179 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:17.254316 (+ 4137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:17.254322 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:17.254327 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:17.254368 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:17.257379 (+ 3011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:17.258285 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:17.258290 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:17.258293 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:17.258471 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:17.259060 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:17.259065 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:17.259069 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:17.259519 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:17.259682 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:17.260556 (+ 874us) server_negotiation.cc:299] Negotiation successful
0928 05:17:17.260786 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 05:17:17.262158 2504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:17.241128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35227 (local address 127.0.97.3:34271)
0928 05:17:17.241904 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:17.241944 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:17.243728 (+ 1784us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:17.244442 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:17.244452 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:17.245141 (+ 689us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:17:17.245902 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:17.245930 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:17.247250 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:17.247256 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:17.249264 (+ 2008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:17.249275 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:17.250279 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:17.250287 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:17.250540 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:17.251171 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:17:17.251195 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:17:17.254107 (+ 2912us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:17:17.257508 (+ 3401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:17.257516 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:17.257529 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:17.258104 (+ 575us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:17.258608 (+ 504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:17.258613 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:17.258617 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:17.258879 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:17.259762 (+ 883us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:17:17.259770 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:17:17.260194 (+ 424us) client_negotiation.cc:769] Sending connection context
0928 05:17:17.260443 (+ 249us) client_negotiation.cc:240] Negotiation successful
0928 05:17:17.260941 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":342,"threads_started":1}
I20260928 05:17:17.280526 2480 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43981
I20260928 05:17:17.280885 2614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43981 every 8 connection(s)
I20260928 05:17:17.281690 2480 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/minicluster-data/ts-2/data/info.pb
I20260928 05:17:17.283813 2615 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35227
I20260928 05:17:17.284031 2615 heartbeater.cc:464] Registering TS with master...
I20260928 05:17:17.284355 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 2480
I20260928 05:17:17.284519 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/minicluster-data/ts-2/wal/instance
I20260928 05:17:17.284651 2179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:17.269906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52301 (local address 127.0.97.62:35227)
0928 05:17:17.270064 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:17.270070 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:17.270843 (+ 773us) server_negotiation.cc:407] Connection header received
0928 05:17:17.271049 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:17.271054 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:17.271141 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:17.271236 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:17.272689 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:17.273656 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:17.275930 (+ 2274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:17.276602 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:17.276748 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:17.278951 (+ 2203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:17.278956 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:17.278960 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:17.278990 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:17.281906 (+ 2916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:17.282679 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:17.282685 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:17.282689 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:17.282850 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:17.283205 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:17.283210 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:17.283213 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:17.283469 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:17.283620 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:17.284055 (+ 435us) server_negotiation.cc:299] Negotiation successful
0928 05:17:17.284243 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 05:17:17.284789 2583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:17.269981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35227 (local address 127.0.97.3:52301)
0928 05:17:17.270726 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:17.270748 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:17.270919 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:17.271396 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:17.271401 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:17.272169 (+ 768us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:17:17.272532 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:17.272544 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:17.273823 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:17.273829 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:17.275722 (+ 1893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:17.275736 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:17.276837 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:17.276847 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:17.277053 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:17.277680 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:17:17.277702 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:17:17.278766 (+ 1064us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:17:17.282037 (+ 3271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:17.282041 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:17.282045 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:17.282575 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:17.282978 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:17.282982 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:17.282985 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:17.283099 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:17.283584 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:17:17.283589 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:17:17.283797 (+ 208us) client_negotiation.cc:769] Sending connection context
0928 05:17:17.283940 (+ 143us) client_negotiation.cc:240] Negotiation successful
0928 05:17:17.284268 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":388,"threads_started":1}
I20260928 05:17:17.284813 2615 heartbeater.cc:511] Master 127.0.97.62:35227 requested a full tablet report, sending...
I20260928 05:17:17.285089 388 external_mini_cluster.cc:1428] Setting key 0dabbb9269dd41b02c99a6e8dcb36a2d
I20260928 05:17:17.286195 2130 ts_manager.cc:194] Registered new tserver with Master: 3de53bd0c5ee445fbfbcb66500285846 (127.0.97.3:43981)
I20260928 05:17:17.287225 2130 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:34271
I20260928 05:17:17.288509 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 05:17:17 dist-test-slave-whsn krb5kdc[2086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572637, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 05:17:17 dist-test-slave-whsn krb5kdc[2086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572637, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:17:17.328023 2179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:17.316130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59592 (local address 127.0.97.62:35227)
0928 05:17:17.316320 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:17.316326 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:17.316580 (+ 254us) server_negotiation.cc:407] Connection header received
0928 05:17:17.316725 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:17.316729 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:17.316788 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:17.316904 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:17.318031 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:17.318739 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:17.319843 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:17.320253 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:17.320338 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:17.323604 (+ 3266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:17.323608 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:17.323612 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:17.323643 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:17.325823 (+ 2180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:17.326367 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:17.326372 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:17.326374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:17.326491 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:17.326812 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:17.326817 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:17.326820 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:17.327056 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:17.327266 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:17.327549 (+ 283us) server_negotiation.cc:299] Negotiation successful
0928 05:17:17.327740 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 05:17:17.331532 2131 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59592:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 05:17:17.338217 2131 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 05:17:17.359140 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:17.348717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43645 (local address 127.0.0.1:57296)
0928 05:17:17.349429 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:17.349493 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:17.349662 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:17.353015 (+ 3353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:17.353020 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:17.353043 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:17:17.353393 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:17.353405 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:17.354827 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:17.354831 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:17.356505 (+ 1674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:17.356519 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:17.358212 (+ 1693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:17.358218 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:17.358397 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:17.358462 (+ 65us) client_negotiation.cc:769] Sending connection context
0928 05:17:17.358561 (+ 99us) client_negotiation.cc:240] Negotiation successful
0928 05:17:17.358726 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"mutex_wait_us":34,"thread_start_us":315,"threads_started":1}
I20260928 05:17:17.359222 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:17.349392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46298 (local address 127.0.97.2:37821)
0928 05:17:17.350729 (+ 1337us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:17.350739 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:17.351405 (+ 666us) server_negotiation.cc:407] Connection header received
0928 05:17:17.351528 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:17.351537 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:17.351781 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:17.351973 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:17:17.352539 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:17.353752 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:17.356704 (+ 2952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:17.358012 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:17.358128 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:17.358321 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:17.358793 (+ 472us) server_negotiation.cc:299] Negotiation successful
0928 05:17:17.358946 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1181,"thread_start_us":1061,"threads_started":1}
I20260928 05:17:17.359582 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:17.348840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37821 (local address 127.0.0.1:46298)
0928 05:17:17.351133 (+ 2293us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:17.351150 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:17.351284 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:17.351987 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:17.351995 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:17.352022 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:17:17.352351 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:17.352362 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:17.353870 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:17.353876 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:17.356511 (+ 2635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:17.356523 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:17.358445 (+ 1922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:17.358451 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:17.358617 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:17.358649 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 05:17:17.358724 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 05:17:17.358823 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2125,"mutex_wait_us":65,"thread_start_us":2011,"threads_started":1}
I20260928 05:17:17.360057 2633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:17.351046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37880 (local address 127.0.97.3:43981)
0928 05:17:17.352769 (+ 1723us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:17.352776 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:17.352801 (+ 25us) server_negotiation.cc:407] Connection header received
0928 05:17:17.352908 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:17.352914 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:17.353111 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:17.353288 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:17:17.354138 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:17.355674 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:17.357661 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:17.358659 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:17.358765 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:17.358962 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:17.359626 (+ 664us) server_negotiation.cc:299] Negotiation successful
0928 05:17:17.359768 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1580,"thread_start_us":2103,"threads_started":1}
I20260928 05:17:17.360574 2632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:17.350498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43981 (local address 127.0.0.1:37880)
0928 05:17:17.351382 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:17.351400 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:17.351509 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:17.353536 (+ 2027us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:17.353540 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:17.353561 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:17:17.353911 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:17.353923 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:17.355824 (+ 1901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:17.355829 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:17.357522 (+ 1693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:17.357532 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:17.359311 (+ 1779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:17.359318 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:17.359478 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:17.359500 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 05:17:17.359579 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 05:17:17.359687 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":772,"mutex_wait_us":27,"thread_start_us":754,"threads_started":1}
I20260928 05:17:17.360599 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:17.348902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57296 (local address 127.0.97.1:43645)
0928 05:17:17.352458 (+ 3556us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:17.352466 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:17.352491 (+ 25us) server_negotiation.cc:407] Connection header received
0928 05:17:17.352576 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:17.352582 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:17.352802 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:17.353570 (+ 768us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:17:17.353744 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:17.354714 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:17.356660 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:17.358011 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:17.359571 (+ 1560us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:17.360145 (+ 574us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:17.360201 (+ 56us) server_negotiation.cc:299] Negotiation successful
0928 05:17:17.360336 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3416,"thread_start_us":3290,"threads_started":1}
I20260928 05:17:17.362764 2406 tablet_service.cc:1467] Processing CreateTablet for tablet 4d8205f90dcb43e8baba55b80a45042b (DEFAULT_TABLE table=test-table [id=2ec14ff82c9d46e7969866c3571459ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:17:17.363505 2267 tablet_service.cc:1467] Processing CreateTablet for tablet 4d8205f90dcb43e8baba55b80a45042b (DEFAULT_TABLE table=test-table [id=2ec14ff82c9d46e7969866c3571459ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:17:17.363922 2406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d8205f90dcb43e8baba55b80a45042b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:17:17.364526 2267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d8205f90dcb43e8baba55b80a45042b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:17:17.367707 2545 tablet_service.cc:1467] Processing CreateTablet for tablet 4d8205f90dcb43e8baba55b80a45042b (DEFAULT_TABLE table=test-table [id=2ec14ff82c9d46e7969866c3571459ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:17:17.368973 2545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d8205f90dcb43e8baba55b80a45042b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:17:17.381193 2634 tablet_bootstrap.cc:492] T 4d8205f90dcb43e8baba55b80a45042b P 64aee4950dd845c6b9174d260e230a36: Bootstrap starting.
I20260928 05:17:17.381420 2337 heartbeater.cc:503] Master 127.0.97.62:35227 was elected leader, sending a full tablet report...
I20260928 05:17:17.382028 2635 tablet_bootstrap.cc:492] T 4d8205f90dcb43e8baba55b80a45042b P af01fe7dfaeb464392cfef10732beb0d: Bootstrap starting.
I20260928 05:17:17.385344 2635 tablet_bootstrap.cc:654] T 4d8205f90dcb43e8baba55b80a45042b P af01fe7dfaeb464392cfef10732beb0d: Neither blocks nor log segments found. Creating new log.
I20260928 05:17:17.386654 2635 log.cc:824] T 4d8205f90dcb43e8baba55b80a45042b P af01fe7dfaeb464392cfef10732beb0d: Log is configured to *not* fsync() on all Append() calls
I20260928 05:17:17.387197 2634 tablet_bootstrap.cc:654] T 4d8205f90dcb43e8baba55b80a45042b P 64aee4950dd845c6b9174d260e230a36: Neither blocks nor log segments found. Creating new log.
I20260928 05:17:17.387532 2636 tablet_bootstrap.cc:492] T 4d8205f90dcb43e8baba55b80a45042b P 3de53bd0c5ee445fbfbcb66500285846: Bootstrap starting.
I20260928 05:17:17.389151 2634 log.cc:824] T 4d8205f90dcb43e8baba55b80a45042b P 64aee4950dd845c6b9174d260e230a36: Log is configured to *not* fsync() on all Append() calls
I20260928 05:17:17.390336 2636 tablet_bootstrap.cc:654] T 4d8205f90dcb43e8baba55b80a45042b P 3de53bd0c5ee445fbfbcb66500285846: Neither blocks nor log segments found. Creating new log.
I20260928 05:17:17.391216 2636 log.cc:824] T 4d8205f90dcb43e8baba55b80a45042b P 3de53bd0c5ee445fbfbcb66500285846: Log is configured to *not* fsync() on all Append() calls
I20260928 05:17:17.392251 2634 tablet_bootstrap.cc:492] T 4d8205f90dcb43e8baba55b80a45042b P 64aee4950dd845c6b9174d260e230a36: No bootstrap required, opened a new log
I20260928 05:17:17.392253 2635 tablet_bootstrap.cc:492] T 4d8205f90dcb43e8baba55b80a45042b P af01fe7dfaeb464392cfef10732beb0d: No bootstrap required, opened a new log
I20260928 05:17:17.392508 2634 ts_tablet_manager.cc:1397] T 4d8205f90dcb43e8baba55b80a45042b P 64aee4950dd845c6b9174d260e230a36: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.000s
I20260928 05:17:17.392508 2635 ts_tablet_manager.cc:1397] T 4d8205f90dcb43e8baba55b80a45042b P af01fe7dfaeb464392cfef10732beb0d: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260928 05:17:17.393415 2636 tablet_bootstrap.cc:492] T 4d8205f90dcb43e8baba55b80a45042b P 3de53bd0c5ee445fbfbcb66500285846: No bootstrap required, opened a new log
I20260928 05:17:17.393623 2636 ts_tablet_manager.cc:1397] T 4d8205f90dcb43e8baba55b80a45042b P 3de53bd0c5ee445fbfbcb66500285846: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260928 05:17:17.400379 2636 raft_consensus.cc:348] T 4d8205f90dcb43e8baba55b80a45042b P 3de53bd0c5ee445fbfbcb66500285846 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64aee4950dd845c6b9174d260e230a36" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43645 } } peers { permanent_uuid: "af01fe7dfaeb464392cfef10732beb0d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37821 } } peers { permanent_uuid: "3de53bd0c5ee445fbfbcb66500285846" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43981 } }
I20260928 05:17:17.400697 2636 raft_consensus.cc:374] T 4d8205f90dcb43e8baba55b80a45042b P 3de53bd0c5ee445fbfbcb66500285846 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:17:17.400784 2636 raft_consensus.cc:729] T 4d8205f90dcb43e8baba55b80a45042b P 3de53bd0c5ee445fbfbcb66500285846 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3de53bd0c5ee445fbfbcb66500285846, State: Initialized, Role: FOLLOWER
I20260928 05:17:17.401332 2636 consensus_queue.cc:260] T 4d8205f90dcb43e8baba55b80a45042b P 3de53bd0c5ee445fbfbcb66500285846 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64aee4950dd845c6b9174d260e230a36" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43645 } } peers { permanent_uuid: "af01fe7dfaeb464392cfef10732beb0d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37821 } } peers { permanent_uuid: "3de53bd0c5ee445fbfbcb66500285846" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43981 } }
I20260928 05:17:17.402256 2615 heartbeater.cc:503] Master 127.0.97.62:35227 was elected leader, sending a full tablet report...
I20260928 05:17:17.402558 2635 raft_consensus.cc:348] T 4d8205f90dcb43e8baba55b80a45042b P af01fe7dfaeb464392cfef10732beb0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64aee4950dd845c6b9174d260e230a36" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43645 } } peers { permanent_uuid: "af01fe7dfaeb464392cfef10732beb0d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37821 } } peers { permanent_uuid: "3de53bd0c5ee445fbfbcb66500285846" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43981 } }
I20260928 05:17:17.402560 2634 raft_consensus.cc:348] T 4d8205f90dcb43e8baba55b80a45042b P 64aee4950dd845c6b9174d260e230a36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64aee4950dd845c6b9174d260e230a36" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43645 } } peers { permanent_uuid: "af01fe7dfaeb464392cfef10732beb0d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37821 } } peers { permanent_uuid: "3de53bd0c5ee445fbfbcb66500285846" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43981 } }
I20260928 05:17:17.402895 2636 ts_tablet_manager.cc:1428] T 4d8205f90dcb43e8baba55b80a45042b P 3de53bd0c5ee445fbfbcb66500285846: Time spent starting tablet: real 0.009s user 0.006s sys 0.003s
I20260928 05:17:17.402928 2634 raft_consensus.cc:374] T 4d8205f90dcb43e8baba55b80a45042b P 64aee4950dd845c6b9174d260e230a36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:17:17.402940 2635 raft_consensus.cc:374] T 4d8205f90dcb43e8baba55b80a45042b P af01fe7dfaeb464392cfef10732beb0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:17:17.403018 2635 raft_consensus.cc:729] T 4d8205f90dcb43e8baba55b80a45042b P af01fe7dfaeb464392cfef10732beb0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af01fe7dfaeb464392cfef10732beb0d, State: Initialized, Role: FOLLOWER
I20260928 05:17:17.403023 2634 raft_consensus.cc:729] T 4d8205f90dcb43e8baba55b80a45042b P 64aee4950dd845c6b9174d260e230a36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64aee4950dd845c6b9174d260e230a36, State: Initialized, Role: FOLLOWER
I20260928 05:17:17.403555 2635 consensus_queue.cc:260] T 4d8205f90dcb43e8baba55b80a45042b P af01fe7dfaeb464392cfef10732beb0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64aee4950dd845c6b9174d260e230a36" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43645 } } peers { permanent_uuid: "af01fe7dfaeb464392cfef10732beb0d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37821 } } peers { permanent_uuid: "3de53bd0c5ee445fbfbcb66500285846" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43981 } }
I20260928 05:17:17.403555 2634 consensus_queue.cc:260] T 4d8205f90dcb43e8baba55b80a45042b P 64aee4950dd845c6b9174d260e230a36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64aee4950dd845c6b9174d260e230a36" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43645 } } peers { permanent_uuid: "af01fe7dfaeb464392cfef10732beb0d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37821 } } peers { permanent_uuid: "3de53bd0c5ee445fbfbcb66500285846" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43981 } }
I20260928 05:17:17.405277 2634 ts_tablet_manager.cc:1428] T 4d8205f90dcb43e8baba55b80a45042b P 64aee4950dd845c6b9174d260e230a36: Time spent starting tablet: real 0.013s user 0.010s sys 0.004s
I20260928 05:17:17.405452 2635 ts_tablet_manager.cc:1428] T 4d8205f90dcb43e8baba55b80a45042b P af01fe7dfaeb464392cfef10732beb0d: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260928 05:17:17.405841 2476 heartbeater.cc:503] Master 127.0.97.62:35227 was elected leader, sending a full tablet report...
I20260928 05:17:17.406522 2641 raft_consensus.cc:482] T 4d8205f90dcb43e8baba55b80a45042b P 64aee4950dd845c6b9174d260e230a36 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:17:17.406736 2641 raft_consensus.cc:504] T 4d8205f90dcb43e8baba55b80a45042b P 64aee4950dd845c6b9174d260e230a36 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64aee4950dd845c6b9174d260e230a36" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43645 } } peers { permanent_uuid: "af01fe7dfaeb464392cfef10732beb0d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37821 } } peers { permanent_uuid: "3de53bd0c5ee445fbfbcb66500285846" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43981 } }
I20260928 05:17:17.408257 2641 leader_election.cc:288] T 4d8205f90dcb43e8baba55b80a45042b P 64aee4950dd845c6b9174d260e230a36 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af01fe7dfaeb464392cfef10732beb0d (127.0.97.2:37821), 3de53bd0c5ee445fbfbcb66500285846 (127.0.97.3:43981)
I20260928 05:17:17.420519 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:17.408819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36215 (local address 127.0.97.2:37821)
0928 05:17:17.409101 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:17.409107 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:17.415587 (+ 6480us) server_negotiation.cc:407] Connection header received
0928 05:17:17.415796 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:17.415801 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:17.415864 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:17.415936 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:17:17.416471 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:17.417211 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:17.418908 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:17.419638 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:17.419751 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:17.419848 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:17.420166 (+ 318us) server_negotiation.cc:299] Negotiation successful
0928 05:17:17.420280 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161}
I20260928 05:17:17.420570 2633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:17.409221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55317 (local address 127.0.97.3:43981)
0928 05:17:17.409372 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:17.409377 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:17.415629 (+ 6252us) server_negotiation.cc:407] Connection header received
0928 05:17:17.415855 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:17.415860 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:17.415925 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:17.416007 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:17:17.416716 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:17.417448 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:17.419048 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:17.419764 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:17.419869 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:17.419960 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:17.420236 (+ 276us) server_negotiation.cc:299] Negotiation successful
0928 05:17:17.420346 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 05:17:17.420650 2644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:17.408814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43981 (local address 127.0.97.1:55317)
0928 05:17:17.415538 (+ 6724us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:17.415557 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:17.415710 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:17.416188 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:17.416193 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:17.416219 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:17:17.416572 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:17.416582 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:17.417571 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:17.417576 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:17.418911 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:17.418922 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:17.419957 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:17.419962 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:17.420108 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:17.420130 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 05:17:17.420201 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 05:17:17.420312 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6589,"thread_start_us":6337,"threads_started":1}
I20260928 05:17:17.420679 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:17.408491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37821 (local address 127.0.97.1:36215)
0928 05:17:17.415482 (+ 6991us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:17.415502 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:17.415664 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:17.415971 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:17.415976 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:17.415998 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:17:17.416312 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:17.416320 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:17.417325 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:17.417330 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:17.418769 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:17.418781 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:17.419828 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:17.419834 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:17.420026 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:17.420049 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 05:17:17.420132 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 05:17:17.420251 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6799,"mutex_wait_us":34,"spinlock_wait_cycles":24832,"thread_start_us":6778,"threads_started":1}
I20260928 05:17:17.421412 2565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d8205f90dcb43e8baba55b80a45042b" candidate_uuid: "64aee4950dd845c6b9174d260e230a36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3de53bd0c5ee445fbfbcb66500285846" is_pre_election: true
I20260928 05:17:17.421458 2426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d8205f90dcb43e8baba55b80a45042b" candidate_uuid: "64aee4950dd845c6b9174d260e230a36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af01fe7dfaeb464392cfef10732beb0d" is_pre_election: true
I20260928 05:17:17.421797 2565 raft_consensus.cc:2454] T 4d8205f90dcb43e8baba55b80a45042b P 3de53bd0c5ee445fbfbcb66500285846 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64aee4950dd845c6b9174d260e230a36 in term 0.
I20260928 05:17:17.421814 2426 raft_consensus.cc:2454] T 4d8205f90dcb43e8baba55b80a45042b P af01fe7dfaeb464392cfef10732beb0d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64aee4950dd845c6b9174d260e230a36 in term 0.
I20260928 05:17:17.422319 2220 leader_election.cc:302] T 4d8205f90dcb43e8baba55b80a45042b P 64aee4950dd845c6b9174d260e230a36 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3de53bd0c5ee445fbfbcb66500285846, 64aee4950dd845c6b9174d260e230a36, af01fe7dfaeb464392cfef10732beb0d; no voters:
I20260928 05:17:17.422652 2643 raft_consensus.cc:2781] T 4d8205f90dcb43e8baba55b80a45042b P 64aee4950dd845c6b9174d260e230a36 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 05:17:17.422791 2643 raft_consensus.cc:482] T 4d8205f90dcb43e8baba55b80a45042b P 64aee4950dd845c6b9174d260e230a36 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 05:17:17.422900 2643 raft_consensus.cc:3037] T 4d8205f90dcb43e8baba55b80a45042b P 64aee4950dd845c6b9174d260e230a36 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:17:17.424935 2643 raft_consensus.cc:504] T 4d8205f90dcb43e8baba55b80a45042b P 64aee4950dd845c6b9174d260e230a36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64aee4950dd845c6b9174d260e230a36" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43645 } } peers { permanent_uuid: "af01fe7dfaeb464392cfef10732beb0d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37821 } } peers { permanent_uuid: "3de53bd0c5ee445fbfbcb66500285846" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43981 } }
I20260928 05:17:17.425522 2643 leader_election.cc:288] T 4d8205f90dcb43e8baba55b80a45042b P 64aee4950dd845c6b9174d260e230a36 [CANDIDATE]: Term 1 election: Requested vote from peers af01fe7dfaeb464392cfef10732beb0d (127.0.97.2:37821), 3de53bd0c5ee445fbfbcb66500285846 (127.0.97.3:43981)
I20260928 05:17:17.425781 2426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d8205f90dcb43e8baba55b80a45042b" candidate_uuid: "64aee4950dd845c6b9174d260e230a36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af01fe7dfaeb464392cfef10732beb0d"
I20260928 05:17:17.425864 2565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d8205f90dcb43e8baba55b80a45042b" candidate_uuid: "64aee4950dd845c6b9174d260e230a36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3de53bd0c5ee445fbfbcb66500285846"
I20260928 05:17:17.425987 2426 raft_consensus.cc:3037] T 4d8205f90dcb43e8baba55b80a45042b P af01fe7dfaeb464392cfef10732beb0d [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:17:17.426020 2565 raft_consensus.cc:3037] T 4d8205f90dcb43e8baba55b80a45042b P 3de53bd0c5ee445fbfbcb66500285846 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:17:17.428931 2565 raft_consensus.cc:2454] T 4d8205f90dcb43e8baba55b80a45042b P 3de53bd0c5ee445fbfbcb66500285846 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64aee4950dd845c6b9174d260e230a36 in term 1.
I20260928 05:17:17.428931 2426 raft_consensus.cc:2454] T 4d8205f90dcb43e8baba55b80a45042b P af01fe7dfaeb464392cfef10732beb0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64aee4950dd845c6b9174d260e230a36 in term 1.
I20260928 05:17:17.429261 2220 leader_election.cc:302] T 4d8205f90dcb43e8baba55b80a45042b P 64aee4950dd845c6b9174d260e230a36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3de53bd0c5ee445fbfbcb66500285846, 64aee4950dd845c6b9174d260e230a36; no voters:
I20260928 05:17:17.429489 2643 raft_consensus.cc:2781] T 4d8205f90dcb43e8baba55b80a45042b P 64aee4950dd845c6b9174d260e230a36 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:17:17.429636 2643 raft_consensus.cc:686] T 4d8205f90dcb43e8baba55b80a45042b P 64aee4950dd845c6b9174d260e230a36 [term 1 LEADER]: Becoming Leader. State: Replica: 64aee4950dd845c6b9174d260e230a36, State: Running, Role: LEADER
I20260928 05:17:17.430023 2643 consensus_queue.cc:237] T 4d8205f90dcb43e8baba55b80a45042b P 64aee4950dd845c6b9174d260e230a36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64aee4950dd845c6b9174d260e230a36" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43645 } } peers { permanent_uuid: "af01fe7dfaeb464392cfef10732beb0d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37821 } } peers { permanent_uuid: "3de53bd0c5ee445fbfbcb66500285846" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43981 } }
I20260928 05:17:17.433897 2130 catalog_manager.cc:5495] T 4d8205f90dcb43e8baba55b80a45042b P 64aee4950dd845c6b9174d260e230a36 reported cstate change: term changed from 0 to 1, leader changed from <none> to 64aee4950dd845c6b9174d260e230a36 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "64aee4950dd845c6b9174d260e230a36" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64aee4950dd845c6b9174d260e230a36" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af01fe7dfaeb464392cfef10732beb0d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3de53bd0c5ee445fbfbcb66500285846" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43981 } health_report { overall_health: UNKNOWN } } }
Sep 28 05:17:17 dist-test-slave-whsn krb5kdc[2086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572635, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:17:17.493773 2179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:17.479841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59596 (local address 127.0.97.62:35227)
0928 05:17:17.480145 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:17.480151 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:17.480618 (+ 467us) server_negotiation.cc:407] Connection header received
0928 05:17:17.480936 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:17.480942 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:17.481004 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:17.481090 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:17.482334 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:17.483116 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:17.484511 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:17.485056 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:17.485250 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:17.488941 (+ 3691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:17.488945 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:17.488949 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:17.488986 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:17.491292 (+ 2306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:17.491894 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:17.491900 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:17.491902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:17.492000 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:17.492333 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:17.492338 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:17.492342 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:17.492575 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:17.492695 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:17.493123 (+ 428us) server_negotiation.cc:299] Negotiation successful
0928 05:17:17.493422 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":118,"server-negotiator.queue_time_us":158}
I20260928 05:17:17.493793 2652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:17.479841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35227 (local address 127.0.0.1:59596)
0928 05:17:17.480510 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:17.480532 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:17.480672 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:17.481141 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:17.481146 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:17.481891 (+ 745us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:17:17.482166 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:17.482175 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:17.483268 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:17.483272 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:17.484338 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:17.484363 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:17.485332 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:17.485342 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:17.485511 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:17.486409 (+ 898us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:17:17.486433 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:17:17.488714 (+ 2281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:17:17.491408 (+ 2694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:17.491420 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:17.491428 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:17.491793 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:17.492106 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:17.492111 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:17.492115 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:17.492243 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:17.492706 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:17:17.492713 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:17:17.492876 (+ 163us) client_negotiation.cc:769] Sending connection context
0928 05:17:17.493037 (+ 161us) client_negotiation.cc:240] Negotiation successful
0928 05:17:17.493292 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":384,"threads_started":1}
I20260928 05:17:17.498176 2130 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59596:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 05:17:17.499076 2130 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 05:17:17.504710 2267 tablet_service.cc:1467] Processing CreateTablet for tablet e6543b8f2a104a4b9658045c13df387b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7d41f399f9cf475a9c109bc1c9c37ce1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 05:17:17.505015 2406 tablet_service.cc:1467] Processing CreateTablet for tablet e6543b8f2a104a4b9658045c13df387b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7d41f399f9cf475a9c109bc1c9c37ce1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 05:17:17.505010 2545 tablet_service.cc:1467] Processing CreateTablet for tablet e6543b8f2a104a4b9658045c13df387b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7d41f399f9cf475a9c109bc1c9c37ce1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 05:17:17.505733 2267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6543b8f2a104a4b9658045c13df387b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:17:17.505879 2545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6543b8f2a104a4b9658045c13df387b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:17:17.505879 2406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6543b8f2a104a4b9658045c13df387b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:17:17.510727 2634 tablet_bootstrap.cc:492] T e6543b8f2a104a4b9658045c13df387b P 64aee4950dd845c6b9174d260e230a36: Bootstrap starting.
I20260928 05:17:17.511933 2634 tablet_bootstrap.cc:654] T e6543b8f2a104a4b9658045c13df387b P 64aee4950dd845c6b9174d260e230a36: Neither blocks nor log segments found. Creating new log.
I20260928 05:17:17.512415 2636 tablet_bootstrap.cc:492] T e6543b8f2a104a4b9658045c13df387b P 3de53bd0c5ee445fbfbcb66500285846: Bootstrap starting.
I20260928 05:17:17.512900 2635 tablet_bootstrap.cc:492] T e6543b8f2a104a4b9658045c13df387b P af01fe7dfaeb464392cfef10732beb0d: Bootstrap starting.
I20260928 05:17:17.513844 2636 tablet_bootstrap.cc:654] T e6543b8f2a104a4b9658045c13df387b P 3de53bd0c5ee445fbfbcb66500285846: Neither blocks nor log segments found. Creating new log.
I20260928 05:17:17.514041 2634 tablet_bootstrap.cc:492] T e6543b8f2a104a4b9658045c13df387b P 64aee4950dd845c6b9174d260e230a36: No bootstrap required, opened a new log
I20260928 05:17:17.514212 2634 ts_tablet_manager.cc:1397] T e6543b8f2a104a4b9658045c13df387b P 64aee4950dd845c6b9174d260e230a36: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260928 05:17:17.514333 2635 tablet_bootstrap.cc:654] T e6543b8f2a104a4b9658045c13df387b P af01fe7dfaeb464392cfef10732beb0d: Neither blocks nor log segments found. Creating new log.
I20260928 05:17:17.515046 2634 raft_consensus.cc:348] T e6543b8f2a104a4b9658045c13df387b P 64aee4950dd845c6b9174d260e230a36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64aee4950dd845c6b9174d260e230a36" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43645 } } peers { permanent_uuid: "af01fe7dfaeb464392cfef10732beb0d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37821 } } peers { permanent_uuid: "3de53bd0c5ee445fbfbcb66500285846" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43981 } }
I20260928 05:17:17.515307 2634 raft_consensus.cc:374] T e6543b8f2a104a4b9658045c13df387b P 64aee4950dd845c6b9174d260e230a36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:17:17.515393 2634 raft_consensus.cc:729] T e6543b8f2a104a4b9658045c13df387b P 64aee4950dd845c6b9174d260e230a36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64aee4950dd845c6b9174d260e230a36, State: Initialized, Role: FOLLOWER
I20260928 05:17:17.515609 2634 consensus_queue.cc:260] T e6543b8f2a104a4b9658045c13df387b P 64aee4950dd845c6b9174d260e230a36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64aee4950dd845c6b9174d260e230a36" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43645 } } peers { permanent_uuid: "af01fe7dfaeb464392cfef10732beb0d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37821 } } peers { permanent_uuid: "3de53bd0c5ee445fbfbcb66500285846" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43981 } }
I20260928 05:17:17.516222 2634 ts_tablet_manager.cc:1428] T e6543b8f2a104a4b9658045c13df387b P 64aee4950dd845c6b9174d260e230a36: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 05:17:17.516098 2643 tablet_replica.cc:439] T e6543b8f2a104a4b9658045c13df387b P 64aee4950dd845c6b9174d260e230a36: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64aee4950dd845c6b9174d260e230a36" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43645 } } peers { permanent_uuid: "af01fe7dfaeb464392cfef10732beb0d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37821 } } peers { permanent_uuid: "3de53bd0c5ee445fbfbcb66500285846" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43981 } } }
I20260928 05:17:17.516381 2643 tablet_replica.cc:442] T e6543b8f2a104a4b9658045c13df387b P 64aee4950dd845c6b9174d260e230a36: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 05:17:17.517743 2635 tablet_bootstrap.cc:492] T e6543b8f2a104a4b9658045c13df387b P af01fe7dfaeb464392cfef10732beb0d: No bootstrap required, opened a new log
I20260928 05:17:17.517881 2635 ts_tablet_manager.cc:1397] T e6543b8f2a104a4b9658045c13df387b P af01fe7dfaeb464392cfef10732beb0d: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260928 05:17:17.518579 2635 raft_consensus.cc:348] T e6543b8f2a104a4b9658045c13df387b P af01fe7dfaeb464392cfef10732beb0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64aee4950dd845c6b9174d260e230a36" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43645 } } peers { permanent_uuid: "af01fe7dfaeb464392cfef10732beb0d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37821 } } peers { permanent_uuid: "3de53bd0c5ee445fbfbcb66500285846" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43981 } }
I20260928 05:17:17.518744 2635 raft_consensus.cc:374] T e6543b8f2a104a4b9658045c13df387b P af01fe7dfaeb464392cfef10732beb0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:17:17.518819 2635 raft_consensus.cc:729] T e6543b8f2a104a4b9658045c13df387b P af01fe7dfaeb464392cfef10732beb0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af01fe7dfaeb464392cfef10732beb0d, State: Initialized, Role: FOLLOWER
I20260928 05:17:17.519011 2635 consensus_queue.cc:260] T e6543b8f2a104a4b9658045c13df387b P af01fe7dfaeb464392cfef10732beb0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64aee4950dd845c6b9174d260e230a36" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43645 } } peers { permanent_uuid: "af01fe7dfaeb464392cfef10732beb0d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37821 } } peers { permanent_uuid: "3de53bd0c5ee445fbfbcb66500285846" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43981 } }
I20260928 05:17:17.519475 2636 tablet_bootstrap.cc:492] T e6543b8f2a104a4b9658045c13df387b P 3de53bd0c5ee445fbfbcb66500285846: No bootstrap required, opened a new log
I20260928 05:17:17.519608 2636 ts_tablet_manager.cc:1397] T e6543b8f2a104a4b9658045c13df387b P 3de53bd0c5ee445fbfbcb66500285846: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260928 05:17:17.519619 2635 ts_tablet_manager.cc:1428] T e6543b8f2a104a4b9658045c13df387b P af01fe7dfaeb464392cfef10732beb0d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260928 05:17:17.519455 2642 tablet_replica.cc:439] T e6543b8f2a104a4b9658045c13df387b P af01fe7dfaeb464392cfef10732beb0d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64aee4950dd845c6b9174d260e230a36" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43645 } } peers { permanent_uuid: "af01fe7dfaeb464392cfef10732beb0d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37821 } } peers { permanent_uuid: "3de53bd0c5ee445fbfbcb66500285846" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43981 } } }
I20260928 05:17:17.519856 2642 tablet_replica.cc:442] T e6543b8f2a104a4b9658045c13df387b P af01fe7dfaeb464392cfef10732beb0d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 05:17:17.520231 2636 raft_consensus.cc:348] T e6543b8f2a104a4b9658045c13df387b P 3de53bd0c5ee445fbfbcb66500285846 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64aee4950dd845c6b9174d260e230a36" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43645 } } peers { permanent_uuid: "af01fe7dfaeb464392cfef10732beb0d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37821 } } peers { permanent_uuid: "3de53bd0c5ee445fbfbcb66500285846" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43981 } }
I20260928 05:17:17.520442 2636 raft_consensus.cc:374] T e6543b8f2a104a4b9658045c13df387b P 3de53bd0c5ee445fbfbcb66500285846 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:17:17.520527 2636 raft_consensus.cc:729] T e6543b8f2a104a4b9658045c13df387b P 3de53bd0c5ee445fbfbcb66500285846 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3de53bd0c5ee445fbfbcb66500285846, State: Initialized, Role: FOLLOWER
I20260928 05:17:17.520740 2636 consensus_queue.cc:260] T e6543b8f2a104a4b9658045c13df387b P 3de53bd0c5ee445fbfbcb66500285846 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64aee4950dd845c6b9174d260e230a36" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43645 } } peers { permanent_uuid: "af01fe7dfaeb464392cfef10732beb0d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37821 } } peers { permanent_uuid: "3de53bd0c5ee445fbfbcb66500285846" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43981 } }
I20260928 05:17:17.521303 2636 ts_tablet_manager.cc:1428] T e6543b8f2a104a4b9658045c13df387b P 3de53bd0c5ee445fbfbcb66500285846: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 05:17:17.521212 2640 tablet_replica.cc:439] T e6543b8f2a104a4b9658045c13df387b P 3de53bd0c5ee445fbfbcb66500285846: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64aee4950dd845c6b9174d260e230a36" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43645 } } peers { permanent_uuid: "af01fe7dfaeb464392cfef10732beb0d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37821 } } peers { permanent_uuid: "3de53bd0c5ee445fbfbcb66500285846" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43981 } } }
I20260928 05:17:17.521548 2640 tablet_replica.cc:442] T e6543b8f2a104a4b9658045c13df387b P 3de53bd0c5ee445fbfbcb66500285846: This TxnStatusTablet replica's current role is: FOLLOWER
W20260928 05:17:17.533527 2616 tablet.cc:2369] T 4d8205f90dcb43e8baba55b80a45042b P 3de53bd0c5ee445fbfbcb66500285846: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 05:17:17.539417 2643 raft_consensus.cc:482] T e6543b8f2a104a4b9658045c13df387b P 64aee4950dd845c6b9174d260e230a36 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:17:17.539577 2643 raft_consensus.cc:504] T e6543b8f2a104a4b9658045c13df387b P 64aee4950dd845c6b9174d260e230a36 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64aee4950dd845c6b9174d260e230a36" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43645 } } peers { permanent_uuid: "af01fe7dfaeb464392cfef10732beb0d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37821 } } peers { permanent_uuid: "3de53bd0c5ee445fbfbcb66500285846" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43981 } }
I20260928 05:17:17.540136 2643 leader_election.cc:288] T e6543b8f2a104a4b9658045c13df387b P 64aee4950dd845c6b9174d260e230a36 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af01fe7dfaeb464392cfef10732beb0d (127.0.97.2:37821), 3de53bd0c5ee445fbfbcb66500285846 (127.0.97.3:43981)
I20260928 05:17:17.540601 2426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6543b8f2a104a4b9658045c13df387b" candidate_uuid: "64aee4950dd845c6b9174d260e230a36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af01fe7dfaeb464392cfef10732beb0d" is_pre_election: true
I20260928 05:17:17.540596 2565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6543b8f2a104a4b9658045c13df387b" candidate_uuid: "64aee4950dd845c6b9174d260e230a36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3de53bd0c5ee445fbfbcb66500285846" is_pre_election: true
I20260928 05:17:17.540820 2426 raft_consensus.cc:2454] T e6543b8f2a104a4b9658045c13df387b P af01fe7dfaeb464392cfef10732beb0d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64aee4950dd845c6b9174d260e230a36 in term 0.
I20260928 05:17:17.540827 2565 raft_consensus.cc:2454] T e6543b8f2a104a4b9658045c13df387b P 3de53bd0c5ee445fbfbcb66500285846 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64aee4950dd845c6b9174d260e230a36 in term 0.
I20260928 05:17:17.541235 2220 leader_election.cc:302] T e6543b8f2a104a4b9658045c13df387b P 64aee4950dd845c6b9174d260e230a36 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3de53bd0c5ee445fbfbcb66500285846, 64aee4950dd845c6b9174d260e230a36; no voters:
I20260928 05:17:17.541504 2643 raft_consensus.cc:2781] T e6543b8f2a104a4b9658045c13df387b P 64aee4950dd845c6b9174d260e230a36 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 05:17:17.541595 2643 raft_consensus.cc:482] T e6543b8f2a104a4b9658045c13df387b P 64aee4950dd845c6b9174d260e230a36 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 05:17:17.541684 2643 raft_consensus.cc:3037] T e6543b8f2a104a4b9658045c13df387b P 64aee4950dd845c6b9174d260e230a36 [term 0 FOLLOWER]: Advancing to term 1
W20260928 05:17:17.572355 2477 tablet.cc:2369] T 4d8205f90dcb43e8baba55b80a45042b P af01fe7dfaeb464392cfef10732beb0d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 05:17:17.572933 2643 raft_consensus.cc:504] T e6543b8f2a104a4b9658045c13df387b P 64aee4950dd845c6b9174d260e230a36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64aee4950dd845c6b9174d260e230a36" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43645 } } peers { permanent_uuid: "af01fe7dfaeb464392cfef10732beb0d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37821 } } peers { permanent_uuid: "3de53bd0c5ee445fbfbcb66500285846" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43981 } }
I20260928 05:17:17.573567 2643 leader_election.cc:288] T e6543b8f2a104a4b9658045c13df387b P 64aee4950dd845c6b9174d260e230a36 [CANDIDATE]: Term 1 election: Requested vote from peers af01fe7dfaeb464392cfef10732beb0d (127.0.97.2:37821), 3de53bd0c5ee445fbfbcb66500285846 (127.0.97.3:43981)
I20260928 05:17:17.573958 2426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6543b8f2a104a4b9658045c13df387b" candidate_uuid: "64aee4950dd845c6b9174d260e230a36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af01fe7dfaeb464392cfef10732beb0d"
I20260928 05:17:17.574096 2565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6543b8f2a104a4b9658045c13df387b" candidate_uuid: "64aee4950dd845c6b9174d260e230a36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3de53bd0c5ee445fbfbcb66500285846"
I20260928 05:17:17.574183 2426 raft_consensus.cc:3037] T e6543b8f2a104a4b9658045c13df387b P af01fe7dfaeb464392cfef10732beb0d [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:17:17.574306 2565 raft_consensus.cc:3037] T e6543b8f2a104a4b9658045c13df387b P 3de53bd0c5ee445fbfbcb66500285846 [term 0 FOLLOWER]: Advancing to term 1
W20260928 05:17:17.624001 2338 tablet.cc:2369] T 4d8205f90dcb43e8baba55b80a45042b P 64aee4950dd845c6b9174d260e230a36: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 05:17:17.655587 2565 raft_consensus.cc:2454] T e6543b8f2a104a4b9658045c13df387b P 3de53bd0c5ee445fbfbcb66500285846 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64aee4950dd845c6b9174d260e230a36 in term 1.
I20260928 05:17:17.655587 2426 raft_consensus.cc:2454] T e6543b8f2a104a4b9658045c13df387b P af01fe7dfaeb464392cfef10732beb0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64aee4950dd845c6b9174d260e230a36 in term 1.
I20260928 05:17:17.656215 2219 leader_election.cc:302] T e6543b8f2a104a4b9658045c13df387b P 64aee4950dd845c6b9174d260e230a36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3de53bd0c5ee445fbfbcb66500285846, 64aee4950dd845c6b9174d260e230a36, af01fe7dfaeb464392cfef10732beb0d; no voters:
I20260928 05:17:17.656594 2643 raft_consensus.cc:2781] T e6543b8f2a104a4b9658045c13df387b P 64aee4950dd845c6b9174d260e230a36 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:17:17.656740 2643 raft_consensus.cc:686] T e6543b8f2a104a4b9658045c13df387b P 64aee4950dd845c6b9174d260e230a36 [term 1 LEADER]: Becoming Leader. State: Replica: 64aee4950dd845c6b9174d260e230a36, State: Running, Role: LEADER
I20260928 05:17:17.657068 2643 consensus_queue.cc:237] T e6543b8f2a104a4b9658045c13df387b P 64aee4950dd845c6b9174d260e230a36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64aee4950dd845c6b9174d260e230a36" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43645 } } peers { permanent_uuid: "af01fe7dfaeb464392cfef10732beb0d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37821 } } peers { permanent_uuid: "3de53bd0c5ee445fbfbcb66500285846" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43981 } }
I20260928 05:17:17.658718 2641 tablet_replica.cc:439] T e6543b8f2a104a4b9658045c13df387b P 64aee4950dd845c6b9174d260e230a36: TxnStatusTablet state changed. Reason: New leader 64aee4950dd845c6b9174d260e230a36. Latest consensus state: current_term: 1 leader_uuid: "64aee4950dd845c6b9174d260e230a36" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64aee4950dd845c6b9174d260e230a36" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43645 } } peers { permanent_uuid: "af01fe7dfaeb464392cfef10732beb0d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37821 } } peers { permanent_uuid: "3de53bd0c5ee445fbfbcb66500285846" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43981 } } }
I20260928 05:17:17.659053 2641 tablet_replica.cc:442] T e6543b8f2a104a4b9658045c13df387b P 64aee4950dd845c6b9174d260e230a36: This TxnStatusTablet replica's current role is: LEADER
I20260928 05:17:17.659832 2657 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 05:17:17.660157 2657 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 05:17:17.660328 2131 catalog_manager.cc:5495] T e6543b8f2a104a4b9658045c13df387b P 64aee4950dd845c6b9174d260e230a36 reported cstate change: term changed from 0 to 1, leader changed from <none> to 64aee4950dd845c6b9174d260e230a36 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "64aee4950dd845c6b9174d260e230a36" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64aee4950dd845c6b9174d260e230a36" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af01fe7dfaeb464392cfef10732beb0d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3de53bd0c5ee445fbfbcb66500285846" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43981 } health_report { overall_health: UNKNOWN } } }
I20260928 05:17:17.677736 2652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:17.672538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43645 (local address 127.0.0.1:57308)
0928 05:17:17.672751 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:17.672772 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:17.672927 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:17.673385 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:17.673390 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:17.673400 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 05:17:17.673575 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:17.673586 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:17.674559 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:17.674565 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:17.675526 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:17.675537 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:17.676211 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:17.676217 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:17.676388 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:17.676437 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 05:17:17.677229 (+ 792us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 05:17:17.677248 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 05:17:17.677325 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 05:17:17.677459 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 05:17:17.677749 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:17.672650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57308 (local address 127.0.97.1:43645)
0928 05:17:17.672935 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:17.672941 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:17.672960 (+ 19us) server_negotiation.cc:407] Connection header received
0928 05:17:17.673142 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:17.673146 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:17.673226 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:17.673370 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:17:17.673698 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:17.674444 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:17.675662 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:17.676076 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:17.676206 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:17.676615 (+ 409us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:17:17.677126 (+ 511us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:17:17.677253 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:17.677358 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 05:17:17.677435 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260928 05:17:17.684768 2565 raft_consensus.cc:1261] T e6543b8f2a104a4b9658045c13df387b P 3de53bd0c5ee445fbfbcb66500285846 [term 1 FOLLOWER]: Refusing update from remote peer 64aee4950dd845c6b9174d260e230a36: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:17:17.684806 2426 raft_consensus.cc:1261] T e6543b8f2a104a4b9658045c13df387b P af01fe7dfaeb464392cfef10732beb0d [term 1 FOLLOWER]: Refusing update from remote peer 64aee4950dd845c6b9174d260e230a36: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:17:17.685818 2643 consensus_queue.cc:1035] T e6543b8f2a104a4b9658045c13df387b P 64aee4950dd845c6b9174d260e230a36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af01fe7dfaeb464392cfef10732beb0d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:17:17.686094 2641 consensus_queue.cc:1035] T e6543b8f2a104a4b9658045c13df387b P 64aee4950dd845c6b9174d260e230a36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3de53bd0c5ee445fbfbcb66500285846" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:17:17.704979 2660 mvcc.cc:205] Tried to move back new op lower bound from 7334185523947499520 to 7334185523845697536. Current Snapshot: MvccSnapshot[applied={T|T < 7334185523947499520}]
I20260928 05:17:17.705804 2640 tablet_replica.cc:439] T e6543b8f2a104a4b9658045c13df387b P 3de53bd0c5ee445fbfbcb66500285846: TxnStatusTablet state changed. Reason: New leader 64aee4950dd845c6b9174d260e230a36. Latest consensus state: current_term: 1 leader_uuid: "64aee4950dd845c6b9174d260e230a36" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64aee4950dd845c6b9174d260e230a36" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43645 } } peers { permanent_uuid: "af01fe7dfaeb464392cfef10732beb0d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37821 } } peers { permanent_uuid: "3de53bd0c5ee445fbfbcb66500285846" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43981 } } }
I20260928 05:17:17.706061 2640 tablet_replica.cc:442] T e6543b8f2a104a4b9658045c13df387b P 3de53bd0c5ee445fbfbcb66500285846: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 05:17:17.707243 2643 tablet_replica.cc:439] T e6543b8f2a104a4b9658045c13df387b P 64aee4950dd845c6b9174d260e230a36: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "64aee4950dd845c6b9174d260e230a36" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64aee4950dd845c6b9174d260e230a36" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43645 } } peers { permanent_uuid: "af01fe7dfaeb464392cfef10732beb0d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37821 } } peers { permanent_uuid: "3de53bd0c5ee445fbfbcb66500285846" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43981 } } }
I20260928 05:17:17.707541 2643 tablet_replica.cc:442] T e6543b8f2a104a4b9658045c13df387b P 64aee4950dd845c6b9174d260e230a36: This TxnStatusTablet replica's current role is: LEADER
I20260928 05:17:17.709609 2661 mvcc.cc:205] Tried to move back new op lower bound from 7334185523947499520 to 7334185523845697536. Current Snapshot: MvccSnapshot[applied={T|T < 7334185523947499520}]
I20260928 05:17:17.710933 2640 tablet_replica.cc:439] T e6543b8f2a104a4b9658045c13df387b P 3de53bd0c5ee445fbfbcb66500285846: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "64aee4950dd845c6b9174d260e230a36" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64aee4950dd845c6b9174d260e230a36" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43645 } } peers { permanent_uuid: "af01fe7dfaeb464392cfef10732beb0d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37821 } } peers { permanent_uuid: "3de53bd0c5ee445fbfbcb66500285846" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43981 } } }
I20260928 05:17:17.711151 2640 tablet_replica.cc:442] T e6543b8f2a104a4b9658045c13df387b P 3de53bd0c5ee445fbfbcb66500285846: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 05:17:17.718107 2642 tablet_replica.cc:439] T e6543b8f2a104a4b9658045c13df387b P af01fe7dfaeb464392cfef10732beb0d: TxnStatusTablet state changed. Reason: New leader 64aee4950dd845c6b9174d260e230a36. Latest consensus state: current_term: 1 leader_uuid: "64aee4950dd845c6b9174d260e230a36" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64aee4950dd845c6b9174d260e230a36" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43645 } } peers { permanent_uuid: "af01fe7dfaeb464392cfef10732beb0d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37821 } } peers { permanent_uuid: "3de53bd0c5ee445fbfbcb66500285846" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43981 } } }
I20260928 05:17:17.718376 2642 tablet_replica.cc:442] T e6543b8f2a104a4b9658045c13df387b P af01fe7dfaeb464392cfef10732beb0d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 05:17:17.719947 2641 tablet_replica.cc:439] T e6543b8f2a104a4b9658045c13df387b P 64aee4950dd845c6b9174d260e230a36: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "64aee4950dd845c6b9174d260e230a36" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64aee4950dd845c6b9174d260e230a36" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43645 } } peers { permanent_uuid: "af01fe7dfaeb464392cfef10732beb0d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37821 } } peers { permanent_uuid: "3de53bd0c5ee445fbfbcb66500285846" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43981 } } }
I20260928 05:17:17.720186 2641 tablet_replica.cc:442] T e6543b8f2a104a4b9658045c13df387b P 64aee4950dd845c6b9174d260e230a36: This TxnStatusTablet replica's current role is: LEADER
I20260928 05:17:17.721601 2662 mvcc.cc:205] Tried to move back new op lower bound from 7334185523947499520 to 7334185523845697536. Current Snapshot: MvccSnapshot[applied={T|T < 7334185523947499520}]
I20260928 05:17:17.722106 2642 tablet_replica.cc:439] T e6543b8f2a104a4b9658045c13df387b P af01fe7dfaeb464392cfef10732beb0d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "64aee4950dd845c6b9174d260e230a36" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64aee4950dd845c6b9174d260e230a36" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43645 } } peers { permanent_uuid: "af01fe7dfaeb464392cfef10732beb0d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37821 } } peers { permanent_uuid: "3de53bd0c5ee445fbfbcb66500285846" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43981 } } }
I20260928 05:17:17.722302 2642 tablet_replica.cc:442] T e6543b8f2a104a4b9658045c13df387b P af01fe7dfaeb464392cfef10732beb0d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 05:17:17.725497 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:17.717009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57324 (local address 127.0.97.1:43645)
0928 05:17:17.717169 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:17.717174 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:17.717193 (+ 19us) server_negotiation.cc:407] Connection header received
0928 05:17:17.717248 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:17.717253 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:17.717315 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:17.717400 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:17:17.717818 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:17.718991 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:17.722524 (+ 3533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:17.723210 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:17.723335 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:17.724720 (+ 1385us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:17:17.724881 (+ 161us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:17:17.724968 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:17.725106 (+ 138us) server_negotiation.cc:299] Negotiation successful
0928 05:17:17.725182 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 05:17:17.742035 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:17.734982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52539 (local address 127.0.97.1:43645)
0928 05:17:17.735159 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:17.735167 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:17.735692 (+ 525us) server_negotiation.cc:407] Connection header received
0928 05:17:17.735822 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:17.735827 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:17.735900 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:17.735992 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:17:17.736445 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:17.737768 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:17.739801 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:17.740490 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:17.740621 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:17.741278 (+ 657us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:17:17.741397 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:17:17.741485 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:17.741650 (+ 165us) server_negotiation.cc:299] Negotiation successful
0928 05:17:17.741722 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 05:17:17.742122 2671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:17.734991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43645 (local address 127.0.97.1:52539)
0928 05:17:17.735606 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:17.735621 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:17.735726 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:17.736010 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:17.736013 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:17.736022 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 05:17:17.736315 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:17.736323 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:17.737892 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:17.737898 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:17.739661 (+ 1763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:17.739674 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:17.740677 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:17.740683 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:17.740842 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:17.740930 (+ 88us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 05:17:17.741499 (+ 569us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 05:17:17.741510 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 05:17:17.741590 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 05:17:17.741722 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":344,"threads_started":1}
I20260928 05:17:17.751345 2565 raft_consensus.cc:1261] T 4d8205f90dcb43e8baba55b80a45042b P 3de53bd0c5ee445fbfbcb66500285846 [term 1 FOLLOWER]: Refusing update from remote peer 64aee4950dd845c6b9174d260e230a36: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:17:17.751345 2426 raft_consensus.cc:1261] T 4d8205f90dcb43e8baba55b80a45042b P af01fe7dfaeb464392cfef10732beb0d [term 1 FOLLOWER]: Refusing update from remote peer 64aee4950dd845c6b9174d260e230a36: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:17:17.751879 2641 consensus_queue.cc:1035] T 4d8205f90dcb43e8baba55b80a45042b P 64aee4950dd845c6b9174d260e230a36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3de53bd0c5ee445fbfbcb66500285846" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:17:17.752130 2643 consensus_queue.cc:1035] T 4d8205f90dcb43e8baba55b80a45042b P 64aee4950dd845c6b9174d260e230a36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af01fe7dfaeb464392cfef10732beb0d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:17:17.812695 2130 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59592:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:17:17.813079 2130 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59592:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:17:17.816383 2130 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aad10d5c9dbc434e9742c4102ad69acd: Sending DeleteTablet for 3 replicas of tablet 4d8205f90dcb43e8baba55b80a45042b
I20260928 05:17:17.817255 2267 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d8205f90dcb43e8baba55b80a45042b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:17:17 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57296
I20260928 05:17:17.817431 2406 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d8205f90dcb43e8baba55b80a45042b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:17:17 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46298
I20260928 05:17:17.817548 2545 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d8205f90dcb43e8baba55b80a45042b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:17:17 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37880
I20260928 05:17:17.821041 2130 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59592:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 05:17:17.822188 2130 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 05:17:17.832989 2676 tablet_replica.cc:330] T 4d8205f90dcb43e8baba55b80a45042b P 64aee4950dd845c6b9174d260e230a36: stopping tablet replica
I20260928 05:17:17.833425 2676 raft_consensus.cc:2229] T 4d8205f90dcb43e8baba55b80a45042b P 64aee4950dd845c6b9174d260e230a36 [term 1 LEADER]: Raft consensus shutting down.
I20260928 05:17:17.834062 2676 raft_consensus.cc:2258] T 4d8205f90dcb43e8baba55b80a45042b P 64aee4950dd845c6b9174d260e230a36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:17:17.838174 2405 tablet_service.cc:1467] Processing CreateTablet for tablet 5b031d17bb274fe480caecea364e840d (DEFAULT_TABLE table=test-table [id=183c744498574edcb2082733c32df066]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:17:17.838709 2405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b031d17bb274fe480caecea364e840d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:17:17.839116 2676 ts_tablet_manager.cc:1905] T 4d8205f90dcb43e8baba55b80a45042b P 64aee4950dd845c6b9174d260e230a36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:17:17.841265 2677 tablet_replica.cc:330] T 4d8205f90dcb43e8baba55b80a45042b P af01fe7dfaeb464392cfef10732beb0d: stopping tablet replica
I20260928 05:17:17.841229 2544 tablet_service.cc:1467] Processing CreateTablet for tablet 5b031d17bb274fe480caecea364e840d (DEFAULT_TABLE table=test-table [id=183c744498574edcb2082733c32df066]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:17:17.841609 2544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b031d17bb274fe480caecea364e840d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:17:17.841609 2677 raft_consensus.cc:2229] T 4d8205f90dcb43e8baba55b80a45042b P af01fe7dfaeb464392cfef10732beb0d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:17:17.841918 2677 raft_consensus.cc:2258] T 4d8205f90dcb43e8baba55b80a45042b P af01fe7dfaeb464392cfef10732beb0d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:17:17.842677 2678 tablet_replica.cc:330] T 4d8205f90dcb43e8baba55b80a45042b P 3de53bd0c5ee445fbfbcb66500285846: stopping tablet replica
I20260928 05:17:17.843037 2678 raft_consensus.cc:2229] T 4d8205f90dcb43e8baba55b80a45042b P 3de53bd0c5ee445fbfbcb66500285846 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:17:17.843318 2678 raft_consensus.cc:2258] T 4d8205f90dcb43e8baba55b80a45042b P 3de53bd0c5ee445fbfbcb66500285846 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:17:17.846817 2677 ts_tablet_manager.cc:1905] T 4d8205f90dcb43e8baba55b80a45042b P af01fe7dfaeb464392cfef10732beb0d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:17:17.848166 2267 tablet_service.cc:1467] Processing CreateTablet for tablet 5b031d17bb274fe480caecea364e840d (DEFAULT_TABLE table=test-table [id=183c744498574edcb2082733c32df066]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:17:17.851692 2636 tablet_bootstrap.cc:492] T 5b031d17bb274fe480caecea364e840d P 3de53bd0c5ee445fbfbcb66500285846: Bootstrap starting.
I20260928 05:17:17.853274 2636 tablet_bootstrap.cc:654] T 5b031d17bb274fe480caecea364e840d P 3de53bd0c5ee445fbfbcb66500285846: Neither blocks nor log segments found. Creating new log.
I20260928 05:17:17.854399 2678 ts_tablet_manager.cc:1905] T 4d8205f90dcb43e8baba55b80a45042b P 3de53bd0c5ee445fbfbcb66500285846: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:17:17.855440 2635 tablet_bootstrap.cc:492] T 5b031d17bb274fe480caecea364e840d P af01fe7dfaeb464392cfef10732beb0d: Bootstrap starting.
I20260928 05:17:17.857069 2635 tablet_bootstrap.cc:654] T 5b031d17bb274fe480caecea364e840d P af01fe7dfaeb464392cfef10732beb0d: Neither blocks nor log segments found. Creating new log.
I20260928 05:17:17.867398 2635 tablet_bootstrap.cc:492] T 5b031d17bb274fe480caecea364e840d P af01fe7dfaeb464392cfef10732beb0d: No bootstrap required, opened a new log
I20260928 05:17:17.867645 2635 ts_tablet_manager.cc:1397] T 5b031d17bb274fe480caecea364e840d P af01fe7dfaeb464392cfef10732beb0d: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.004s
I20260928 05:17:17.869081 2635 raft_consensus.cc:348] T 5b031d17bb274fe480caecea364e840d P af01fe7dfaeb464392cfef10732beb0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64aee4950dd845c6b9174d260e230a36" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43645 } } peers { permanent_uuid: "af01fe7dfaeb464392cfef10732beb0d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37821 } } peers { permanent_uuid: "3de53bd0c5ee445fbfbcb66500285846" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43981 } }
I20260928 05:17:17.869475 2635 raft_consensus.cc:374] T 5b031d17bb274fe480caecea364e840d P af01fe7dfaeb464392cfef10732beb0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:17:17.869597 2635 raft_consensus.cc:729] T 5b031d17bb274fe480caecea364e840d P af01fe7dfaeb464392cfef10732beb0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af01fe7dfaeb464392cfef10732beb0d, State: Initialized, Role: FOLLOWER
I20260928 05:17:17.869976 2635 consensus_queue.cc:260] T 5b031d17bb274fe480caecea364e840d P af01fe7dfaeb464392cfef10732beb0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64aee4950dd845c6b9174d260e230a36" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43645 } } peers { permanent_uuid: "af01fe7dfaeb464392cfef10732beb0d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37821 } } peers { permanent_uuid: "3de53bd0c5ee445fbfbcb66500285846" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43981 } }
I20260928 05:17:17.873018 2635 ts_tablet_manager.cc:1428] T 5b031d17bb274fe480caecea364e840d P af01fe7dfaeb464392cfef10732beb0d: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260928 05:17:17.876590 2636 tablet_bootstrap.cc:492] T 5b031d17bb274fe480caecea364e840d P 3de53bd0c5ee445fbfbcb66500285846: No bootstrap required, opened a new log
I20260928 05:17:17.876736 2636 ts_tablet_manager.cc:1397] T 5b031d17bb274fe480caecea364e840d P 3de53bd0c5ee445fbfbcb66500285846: Time spent bootstrapping tablet: real 0.025s user 0.017s sys 0.007s
I20260928 05:17:17.877331 2636 raft_consensus.cc:348] T 5b031d17bb274fe480caecea364e840d P 3de53bd0c5ee445fbfbcb66500285846 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64aee4950dd845c6b9174d260e230a36" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43645 } } peers { permanent_uuid: "af01fe7dfaeb464392cfef10732beb0d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37821 } } peers { permanent_uuid: "3de53bd0c5ee445fbfbcb66500285846" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43981 } }
I20260928 05:17:17.877511 2636 raft_consensus.cc:374] T 5b031d17bb274fe480caecea364e840d P 3de53bd0c5ee445fbfbcb66500285846 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:17:17.877584 2636 raft_consensus.cc:729] T 5b031d17bb274fe480caecea364e840d P 3de53bd0c5ee445fbfbcb66500285846 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3de53bd0c5ee445fbfbcb66500285846, State: Initialized, Role: FOLLOWER
I20260928 05:17:17.877797 2636 consensus_queue.cc:260] T 5b031d17bb274fe480caecea364e840d P 3de53bd0c5ee445fbfbcb66500285846 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64aee4950dd845c6b9174d260e230a36" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43645 } } peers { permanent_uuid: "af01fe7dfaeb464392cfef10732beb0d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37821 } } peers { permanent_uuid: "3de53bd0c5ee445fbfbcb66500285846" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43981 } }
I20260928 05:17:17.878223 2636 ts_tablet_manager.cc:1428] T 5b031d17bb274fe480caecea364e840d P 3de53bd0c5ee445fbfbcb66500285846: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 05:17:17.919148 2267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b031d17bb274fe480caecea364e840d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:17:17.922317 2677 ts_tablet_manager.cc:1918] T 4d8205f90dcb43e8baba55b80a45042b P af01fe7dfaeb464392cfef10732beb0d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 05:17:17.922540 2677 log.cc:1196] T 4d8205f90dcb43e8baba55b80a45042b P af01fe7dfaeb464392cfef10732beb0d: Deleting WAL directory at /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/minicluster-data/ts-1/wal/wals/4d8205f90dcb43e8baba55b80a45042b
I20260928 05:17:17.923106 2677 ts_tablet_manager.cc:1939] T 4d8205f90dcb43e8baba55b80a45042b P af01fe7dfaeb464392cfef10732beb0d: Deleting consensus metadata
I20260928 05:17:18.005824 2116 catalog_manager.cc:4841] TS af01fe7dfaeb464392cfef10732beb0d (127.0.97.2:37821): tablet 4d8205f90dcb43e8baba55b80a45042b (table test-table [id=2ec14ff82c9d46e7969866c3571459ee]) successfully deleted
I20260928 05:17:18.020030 2678 ts_tablet_manager.cc:1918] T 4d8205f90dcb43e8baba55b80a45042b P 3de53bd0c5ee445fbfbcb66500285846: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 05:17:18.020028 2676 ts_tablet_manager.cc:1918] T 4d8205f90dcb43e8baba55b80a45042b P 64aee4950dd845c6b9174d260e230a36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 05:17:18.020217 2678 log.cc:1196] T 4d8205f90dcb43e8baba55b80a45042b P 3de53bd0c5ee445fbfbcb66500285846: Deleting WAL directory at /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/minicluster-data/ts-2/wal/wals/4d8205f90dcb43e8baba55b80a45042b
I20260928 05:17:18.020224 2676 log.cc:1196] T 4d8205f90dcb43e8baba55b80a45042b P 64aee4950dd845c6b9174d260e230a36: Deleting WAL directory at /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/minicluster-data/ts-0/wal/wals/4d8205f90dcb43e8baba55b80a45042b
I20260928 05:17:18.020714 2678 ts_tablet_manager.cc:1939] T 4d8205f90dcb43e8baba55b80a45042b P 3de53bd0c5ee445fbfbcb66500285846: Deleting consensus metadata
I20260928 05:17:18.020720 2676 ts_tablet_manager.cc:1939] T 4d8205f90dcb43e8baba55b80a45042b P 64aee4950dd845c6b9174d260e230a36: Deleting consensus metadata
I20260928 05:17:18.022531 2115 catalog_manager.cc:4841] TS 64aee4950dd845c6b9174d260e230a36 (127.0.97.1:43645): tablet 4d8205f90dcb43e8baba55b80a45042b (table test-table [id=2ec14ff82c9d46e7969866c3571459ee]) successfully deleted
I20260928 05:17:18.023182 2117 catalog_manager.cc:4841] TS 3de53bd0c5ee445fbfbcb66500285846 (127.0.97.3:43981): tablet 4d8205f90dcb43e8baba55b80a45042b (table test-table [id=2ec14ff82c9d46e7969866c3571459ee]) successfully deleted
I20260928 05:17:18.025346 2681 tablet_bootstrap.cc:492] T 5b031d17bb274fe480caecea364e840d P 64aee4950dd845c6b9174d260e230a36: Bootstrap starting.
I20260928 05:17:18.026685 2681 tablet_bootstrap.cc:654] T 5b031d17bb274fe480caecea364e840d P 64aee4950dd845c6b9174d260e230a36: Neither blocks nor log segments found. Creating new log.
I20260928 05:17:18.028461 2681 tablet_bootstrap.cc:492] T 5b031d17bb274fe480caecea364e840d P 64aee4950dd845c6b9174d260e230a36: No bootstrap required, opened a new log
I20260928 05:17:18.028580 2681 ts_tablet_manager.cc:1397] T 5b031d17bb274fe480caecea364e840d P 64aee4950dd845c6b9174d260e230a36: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 05:17:18.029258 2681 raft_consensus.cc:348] T 5b031d17bb274fe480caecea364e840d P 64aee4950dd845c6b9174d260e230a36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64aee4950dd845c6b9174d260e230a36" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43645 } } peers { permanent_uuid: "af01fe7dfaeb464392cfef10732beb0d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37821 } } peers { permanent_uuid: "3de53bd0c5ee445fbfbcb66500285846" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43981 } }
I20260928 05:17:18.029453 2681 raft_consensus.cc:374] T 5b031d17bb274fe480caecea364e840d P 64aee4950dd845c6b9174d260e230a36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:17:18.029538 2681 raft_consensus.cc:729] T 5b031d17bb274fe480caecea364e840d P 64aee4950dd845c6b9174d260e230a36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64aee4950dd845c6b9174d260e230a36, State: Initialized, Role: FOLLOWER
I20260928 05:17:18.029753 2681 consensus_queue.cc:260] T 5b031d17bb274fe480caecea364e840d P 64aee4950dd845c6b9174d260e230a36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64aee4950dd845c6b9174d260e230a36" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43645 } } peers { permanent_uuid: "af01fe7dfaeb464392cfef10732beb0d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37821 } } peers { permanent_uuid: "3de53bd0c5ee445fbfbcb66500285846" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43981 } }
I20260928 05:17:18.030244 2681 ts_tablet_manager.cc:1428] T 5b031d17bb274fe480caecea364e840d P 64aee4950dd845c6b9174d260e230a36: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260928 05:17:18.045440 2640 raft_consensus.cc:482] T 5b031d17bb274fe480caecea364e840d P 3de53bd0c5ee445fbfbcb66500285846 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:17:18.045689 2640 raft_consensus.cc:504] T 5b031d17bb274fe480caecea364e840d P 3de53bd0c5ee445fbfbcb66500285846 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64aee4950dd845c6b9174d260e230a36" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43645 } } peers { permanent_uuid: "af01fe7dfaeb464392cfef10732beb0d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37821 } } peers { permanent_uuid: "3de53bd0c5ee445fbfbcb66500285846" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43981 } }
I20260928 05:17:18.047669 2640 leader_election.cc:288] T 5b031d17bb274fe480caecea364e840d P 3de53bd0c5ee445fbfbcb66500285846 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64aee4950dd845c6b9174d260e230a36 (127.0.97.1:43645), af01fe7dfaeb464392cfef10732beb0d (127.0.97.2:37821)
I20260928 05:17:18.067072 2685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:18.048058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33943 (local address 127.0.97.2:37821)
0928 05:17:18.060330 (+ 12272us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:18.060336 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:18.061232 (+ 896us) server_negotiation.cc:407] Connection header received
0928 05:17:18.061436 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:18.061442 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:18.061513 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:18.061640 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:17:18.062167 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:18.063278 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:18.065491 (+ 2213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:18.066213 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:18.066317 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:18.066407 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:18.066657 (+ 250us) server_negotiation.cc:299] Negotiation successful
0928 05:17:18.066785 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12166,"thread_start_us":11865,"threads_started":1}
I20260928 05:17:18.067107 2684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:18.047961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37821 (local address 127.0.97.3:33943)
0928 05:17:18.061150 (+ 13189us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:18.061165 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:18.061302 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:18.061652 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:18.061657 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:18.061689 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:17:18.061958 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:18.061979 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:18.063408 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:18.063414 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:18.065353 (+ 1939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:18.065364 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:18.066344 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:18.066349 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:18.066515 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:18.066537 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 05:17:18.066623 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 05:17:18.066742 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13080,"mutex_wait_us":1,"spinlock_wait_cycles":1280,"thread_start_us":12952,"threads_started":1}
I20260928 05:17:18.067386 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:18.048228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42209 (local address 127.0.97.1:43645)
0928 05:17:18.048405 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:18.048412 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:18.060424 (+ 12012us) server_negotiation.cc:407] Connection header received
0928 05:17:18.060678 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:18.060684 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:18.060762 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:18.060896 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:17:18.061625 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:18.062590 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:18.064905 (+ 2315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:18.066245 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:18.066395 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:18.066490 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:18.066988 (+ 498us) server_negotiation.cc:299] Negotiation successful
0928 05:17:18.067116 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 05:17:18.067731 2426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b031d17bb274fe480caecea364e840d" candidate_uuid: "3de53bd0c5ee445fbfbcb66500285846" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af01fe7dfaeb464392cfef10732beb0d" is_pre_election: true
I20260928 05:17:18.067938 2426 raft_consensus.cc:2454] T 5b031d17bb274fe480caecea364e840d P af01fe7dfaeb464392cfef10732beb0d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3de53bd0c5ee445fbfbcb66500285846 in term 0.
I20260928 05:17:18.068368 2497 leader_election.cc:302] T 5b031d17bb274fe480caecea364e840d P 3de53bd0c5ee445fbfbcb66500285846 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3de53bd0c5ee445fbfbcb66500285846, af01fe7dfaeb464392cfef10732beb0d; no voters:
I20260928 05:17:18.068464 2683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:18.047835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43645 (local address 127.0.97.3:42209)
0928 05:17:18.060307 (+ 12472us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:18.060326 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:18.060514 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:18.061152 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:18.061156 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:18.061183 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:17:18.061476 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:18.061486 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:18.062711 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:18.062716 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:18.064728 (+ 2012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:18.064743 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:18.066621 (+ 1878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:18.066627 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:18.066807 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:18.066844 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 05:17:18.067942 (+ 1098us) client_negotiation.cc:240] Negotiation successful
0928 05:17:18.068085 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12348,"mutex_wait_us":42,"spinlock_wait_cycles":17664,"thread_start_us":12767,"threads_started":1}
I20260928 05:17:18.068768 2640 raft_consensus.cc:2781] T 5b031d17bb274fe480caecea364e840d P 3de53bd0c5ee445fbfbcb66500285846 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 05:17:18.068902 2640 raft_consensus.cc:482] T 5b031d17bb274fe480caecea364e840d P 3de53bd0c5ee445fbfbcb66500285846 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 05:17:18.069017 2640 raft_consensus.cc:3037] T 5b031d17bb274fe480caecea364e840d P 3de53bd0c5ee445fbfbcb66500285846 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:17:18.072283 2287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b031d17bb274fe480caecea364e840d" candidate_uuid: "3de53bd0c5ee445fbfbcb66500285846" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64aee4950dd845c6b9174d260e230a36" is_pre_election: true
I20260928 05:17:18.072649 2287 raft_consensus.cc:2454] T 5b031d17bb274fe480caecea364e840d P 64aee4950dd845c6b9174d260e230a36 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3de53bd0c5ee445fbfbcb66500285846 in term 0.
I20260928 05:17:18.121228 2640 raft_consensus.cc:504] T 5b031d17bb274fe480caecea364e840d P 3de53bd0c5ee445fbfbcb66500285846 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64aee4950dd845c6b9174d260e230a36" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43645 } } peers { permanent_uuid: "af01fe7dfaeb464392cfef10732beb0d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37821 } } peers { permanent_uuid: "3de53bd0c5ee445fbfbcb66500285846" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43981 } }
I20260928 05:17:18.121914 2640 leader_election.cc:288] T 5b031d17bb274fe480caecea364e840d P 3de53bd0c5ee445fbfbcb66500285846 [CANDIDATE]: Term 1 election: Requested vote from peers 64aee4950dd845c6b9174d260e230a36 (127.0.97.1:43645), af01fe7dfaeb464392cfef10732beb0d (127.0.97.2:37821)
I20260928 05:17:18.122572 2287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b031d17bb274fe480caecea364e840d" candidate_uuid: "3de53bd0c5ee445fbfbcb66500285846" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64aee4950dd845c6b9174d260e230a36"
I20260928 05:17:18.122570 2426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b031d17bb274fe480caecea364e840d" candidate_uuid: "3de53bd0c5ee445fbfbcb66500285846" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af01fe7dfaeb464392cfef10732beb0d"
I20260928 05:17:18.122785 2426 raft_consensus.cc:3037] T 5b031d17bb274fe480caecea364e840d P af01fe7dfaeb464392cfef10732beb0d [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:17:18.122794 2287 raft_consensus.cc:3037] T 5b031d17bb274fe480caecea364e840d P 64aee4950dd845c6b9174d260e230a36 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:17:18.224243 2426 raft_consensus.cc:2454] T 5b031d17bb274fe480caecea364e840d P af01fe7dfaeb464392cfef10732beb0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3de53bd0c5ee445fbfbcb66500285846 in term 1.
I20260928 05:17:18.224246 2287 raft_consensus.cc:2454] T 5b031d17bb274fe480caecea364e840d P 64aee4950dd845c6b9174d260e230a36 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3de53bd0c5ee445fbfbcb66500285846 in term 1.
I20260928 05:17:18.224836 2496 leader_election.cc:302] T 5b031d17bb274fe480caecea364e840d P 3de53bd0c5ee445fbfbcb66500285846 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3de53bd0c5ee445fbfbcb66500285846, 64aee4950dd845c6b9174d260e230a36, af01fe7dfaeb464392cfef10732beb0d; no voters:
I20260928 05:17:18.225219 2640 raft_consensus.cc:2781] T 5b031d17bb274fe480caecea364e840d P 3de53bd0c5ee445fbfbcb66500285846 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:17:18.225847 2640 raft_consensus.cc:686] T 5b031d17bb274fe480caecea364e840d P 3de53bd0c5ee445fbfbcb66500285846 [term 1 LEADER]: Becoming Leader. State: Replica: 3de53bd0c5ee445fbfbcb66500285846, State: Running, Role: LEADER
I20260928 05:17:18.226256 2640 consensus_queue.cc:237] T 5b031d17bb274fe480caecea364e840d P 3de53bd0c5ee445fbfbcb66500285846 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64aee4950dd845c6b9174d260e230a36" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43645 } } peers { permanent_uuid: "af01fe7dfaeb464392cfef10732beb0d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37821 } } peers { permanent_uuid: "3de53bd0c5ee445fbfbcb66500285846" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43981 } }
I20260928 05:17:18.229662 2131 catalog_manager.cc:5495] T 5b031d17bb274fe480caecea364e840d P 3de53bd0c5ee445fbfbcb66500285846 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3de53bd0c5ee445fbfbcb66500285846 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "3de53bd0c5ee445fbfbcb66500285846" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64aee4950dd845c6b9174d260e230a36" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af01fe7dfaeb464392cfef10732beb0d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3de53bd0c5ee445fbfbcb66500285846" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43981 } health_report { overall_health: HEALTHY } } }
I20260928 05:17:18.335558 2697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:18.329826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37894 (local address 127.0.97.3:43981)
0928 05:17:18.330732 (+ 906us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:18.330736 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:18.330753 (+ 17us) server_negotiation.cc:407] Connection header received
0928 05:17:18.330830 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:18.330833 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:18.330898 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:18.331035 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:17:18.331446 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:18.332194 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:18.333525 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:18.333922 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:18.334022 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:18.334414 (+ 392us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:17:18.334956 (+ 542us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:17:18.335088 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:18.335191 (+ 103us) server_negotiation.cc:299] Negotiation successful
0928 05:17:18.335248 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":802,"thread_start_us":689,"threads_started":1}
I20260928 05:17:18.354123 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:18.344552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52671 (local address 127.0.97.1:43645)
0928 05:17:18.345099 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:18.345109 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:18.348768 (+ 3659us) server_negotiation.cc:407] Connection header received
0928 05:17:18.348988 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:18.348993 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:18.349077 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:18.349169 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:17:18.349648 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:18.350920 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:18.352168 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:18.352814 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:18.352977 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:18.353334 (+ 357us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:17:18.353522 (+ 188us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:17:18.353612 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:18.353732 (+ 120us) server_negotiation.cc:299] Negotiation successful
0928 05:17:18.353801 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 05:17:18.354146 2700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:18.344456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43645 (local address 127.0.97.3:52671)
0928 05:17:18.348684 (+ 4228us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:18.348699 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:18.348828 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:18.349201 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:18.349205 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:18.349216 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 05:17:18.349478 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:18.349488 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:18.351034 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:18.351038 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:18.352040 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:18.352050 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:18.352999 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:18.353003 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:18.353114 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:18.353167 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 05:17:18.353616 (+ 449us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 05:17:18.353622 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 05:17:18.353679 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 05:17:18.353802 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4076,"thread_start_us":3938,"threads_started":1}
I20260928 05:17:18.371121 2287 raft_consensus.cc:1261] T 5b031d17bb274fe480caecea364e840d P 64aee4950dd845c6b9174d260e230a36 [term 1 FOLLOWER]: Refusing update from remote peer 3de53bd0c5ee445fbfbcb66500285846: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:17:18.371841 2686 consensus_queue.cc:1035] T 5b031d17bb274fe480caecea364e840d P 3de53bd0c5ee445fbfbcb66500285846 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64aee4950dd845c6b9174d260e230a36" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:17:18.376910 2426 raft_consensus.cc:1261] T 5b031d17bb274fe480caecea364e840d P af01fe7dfaeb464392cfef10732beb0d [term 1 FOLLOWER]: Refusing update from remote peer 3de53bd0c5ee445fbfbcb66500285846: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:17:18.377457 2686 consensus_queue.cc:1035] T 5b031d17bb274fe480caecea364e840d P 3de53bd0c5ee445fbfbcb66500285846 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af01fe7dfaeb464392cfef10732beb0d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:17:18.415078 2697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:18.409090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52555 (local address 127.0.97.3:43981)
0928 05:17:18.409236 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:18.409242 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:18.409923 (+ 681us) server_negotiation.cc:407] Connection header received
0928 05:17:18.410088 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:18.410093 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:18.410173 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:18.410267 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:17:18.410698 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:18.411663 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:18.413501 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:18.413933 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:18.414035 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:18.414446 (+ 411us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:17:18.414542 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:17:18.414610 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:18.414740 (+ 130us) server_negotiation.cc:299] Negotiation successful
0928 05:17:18.414808 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 05:17:18.415252 2706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:18.409046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43981 (local address 127.0.97.1:52555)
0928 05:17:18.409800 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:18.409829 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:18.409951 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:18.410285 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:18.410288 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:18.410298 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 05:17:18.410544 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:18.410552 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:18.411792 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:18.411797 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:18.413366 (+ 1569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:18.413380 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:18.414094 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:18.414100 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:18.414236 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:18.414283 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 05:17:18.414635 (+ 352us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 05:17:18.414645 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 05:17:18.414715 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 05:17:18.414860 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":336,"threads_started":1}
W20260928 05:17:18.416551 2689 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 05:17:20.013756 2130 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59592:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:17:20.013962 2130 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59592:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:17:20.017514 2130 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aad10d5c9dbc434e9742c4102ad69acd: Sending DeleteTablet for 3 replicas of tablet 5b031d17bb274fe480caecea364e840d
I20260928 05:17:20.018211 2545 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b031d17bb274fe480caecea364e840d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:17:20 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37880
I20260928 05:17:20.018216 2267 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b031d17bb274fe480caecea364e840d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:17:20 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57296
I20260928 05:17:20.018275 2405 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b031d17bb274fe480caecea364e840d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:17:20 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46298
I20260928 05:17:20.030098 2734 tablet_replica.cc:330] T 5b031d17bb274fe480caecea364e840d P af01fe7dfaeb464392cfef10732beb0d: stopping tablet replica
I20260928 05:17:20.030208 2732 tablet_replica.cc:330] T 5b031d17bb274fe480caecea364e840d P 3de53bd0c5ee445fbfbcb66500285846: stopping tablet replica
I20260928 05:17:20.030332 2734 raft_consensus.cc:2229] T 5b031d17bb274fe480caecea364e840d P af01fe7dfaeb464392cfef10732beb0d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:17:20.030475 2732 raft_consensus.cc:2229] T 5b031d17bb274fe480caecea364e840d P 3de53bd0c5ee445fbfbcb66500285846 [term 1 LEADER]: Raft consensus shutting down.
I20260928 05:17:20.030614 2734 raft_consensus.cc:2258] T 5b031d17bb274fe480caecea364e840d P af01fe7dfaeb464392cfef10732beb0d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:17:20.031056 2732 raft_consensus.cc:2258] T 5b031d17bb274fe480caecea364e840d P 3de53bd0c5ee445fbfbcb66500285846 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:17:20.031112 2733 tablet_replica.cc:330] T 5b031d17bb274fe480caecea364e840d P 64aee4950dd845c6b9174d260e230a36: stopping tablet replica
I20260928 05:17:20.031764 2734 ts_tablet_manager.cc:1905] T 5b031d17bb274fe480caecea364e840d P af01fe7dfaeb464392cfef10732beb0d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:17:20.031796 2732 ts_tablet_manager.cc:1905] T 5b031d17bb274fe480caecea364e840d P 3de53bd0c5ee445fbfbcb66500285846: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:17:20.033039 2733 raft_consensus.cc:2229] T 5b031d17bb274fe480caecea364e840d P 64aee4950dd845c6b9174d260e230a36 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:17:20.033300 2733 raft_consensus.cc:2258] T 5b031d17bb274fe480caecea364e840d P 64aee4950dd845c6b9174d260e230a36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:17:20.034227 2733 ts_tablet_manager.cc:1905] T 5b031d17bb274fe480caecea364e840d P 64aee4950dd845c6b9174d260e230a36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:17:20.037443 2732 ts_tablet_manager.cc:1918] T 5b031d17bb274fe480caecea364e840d P 3de53bd0c5ee445fbfbcb66500285846: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 05:17:20.037609 2732 log.cc:1196] T 5b031d17bb274fe480caecea364e840d P 3de53bd0c5ee445fbfbcb66500285846: Deleting WAL directory at /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/minicluster-data/ts-2/wal/wals/5b031d17bb274fe480caecea364e840d
I20260928 05:17:20.037729 2734 ts_tablet_manager.cc:1918] T 5b031d17bb274fe480caecea364e840d P af01fe7dfaeb464392cfef10732beb0d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 05:17:20.037935 2734 log.cc:1196] T 5b031d17bb274fe480caecea364e840d P af01fe7dfaeb464392cfef10732beb0d: Deleting WAL directory at /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/minicluster-data/ts-1/wal/wals/5b031d17bb274fe480caecea364e840d
I20260928 05:17:20.038094 2732 ts_tablet_manager.cc:1939] T 5b031d17bb274fe480caecea364e840d P 3de53bd0c5ee445fbfbcb66500285846: Deleting consensus metadata
I20260928 05:17:20.038359 2734 ts_tablet_manager.cc:1939] T 5b031d17bb274fe480caecea364e840d P af01fe7dfaeb464392cfef10732beb0d: Deleting consensus metadata
I20260928 05:17:20.039067 2117 catalog_manager.cc:4841] TS 3de53bd0c5ee445fbfbcb66500285846 (127.0.97.3:43981): tablet 5b031d17bb274fe480caecea364e840d (table test-table [id=183c744498574edcb2082733c32df066]) successfully deleted
I20260928 05:17:20.039462 2116 catalog_manager.cc:4841] TS af01fe7dfaeb464392cfef10732beb0d (127.0.97.2:37821): tablet 5b031d17bb274fe480caecea364e840d (table test-table [id=183c744498574edcb2082733c32df066]) successfully deleted
I20260928 05:17:20.040179 2733 ts_tablet_manager.cc:1918] T 5b031d17bb274fe480caecea364e840d P 64aee4950dd845c6b9174d260e230a36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 05:17:20.040269 2733 log.cc:1196] T 5b031d17bb274fe480caecea364e840d P 64aee4950dd845c6b9174d260e230a36: Deleting WAL directory at /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/minicluster-data/ts-0/wal/wals/5b031d17bb274fe480caecea364e840d
I20260928 05:17:20.040578 2733 ts_tablet_manager.cc:1939] T 5b031d17bb274fe480caecea364e840d P 64aee4950dd845c6b9174d260e230a36: Deleting consensus metadata
I20260928 05:17:20.041451 2115 catalog_manager.cc:4841] TS 64aee4950dd845c6b9174d260e230a36 (127.0.97.1:43645): tablet 5b031d17bb274fe480caecea364e840d (table test-table [id=183c744498574edcb2082733c32df066]) successfully deleted
Sep 28 05:17:20 dist-test-slave-whsn krb5kdc[2086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572640, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 05:17:20 dist-test-slave-whsn krb5kdc[2086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572640, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:17:20.067344 2742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:20.054344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59602 (local address 127.0.97.62:35227)
0928 05:17:20.054827 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:20.054832 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:20.054851 (+ 19us) server_negotiation.cc:407] Connection header received
0928 05:17:20.054991 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:20.054995 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:20.055052 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:20.055130 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:20.056657 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:20.057568 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:20.058692 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:20.059082 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:20.059169 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:20.062658 (+ 3489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:20.062663 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:20.062666 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:20.062694 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:20.065105 (+ 2411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:20.065650 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:20.065655 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:20.065659 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:20.065765 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:20.066091 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:20.066096 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:20.066099 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:20.066340 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:20.066544 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:20.066847 (+ 303us) server_negotiation.cc:299] Negotiation successful
0928 05:17:20.067032 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":241,"threads_started":1}
I20260928 05:17:20.070209 2130 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59602:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 05:17:20.070756 2130 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 05:17:20.076421 2267 tablet_service.cc:1467] Processing CreateTablet for tablet f9057d80b6a64f01937179121f461a1f (DEFAULT_TABLE table=test-table [id=0061caf983a14ac7aaba65d43c654e82]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:17:20.076453 2405 tablet_service.cc:1467] Processing CreateTablet for tablet f9057d80b6a64f01937179121f461a1f (DEFAULT_TABLE table=test-table [id=0061caf983a14ac7aaba65d43c654e82]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:17:20.076658 2545 tablet_service.cc:1467] Processing CreateTablet for tablet f9057d80b6a64f01937179121f461a1f (DEFAULT_TABLE table=test-table [id=0061caf983a14ac7aaba65d43c654e82]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:17:20.076829 2267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9057d80b6a64f01937179121f461a1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:17:20.076887 2405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9057d80b6a64f01937179121f461a1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:17:20.077036 2545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9057d80b6a64f01937179121f461a1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:17:20.082552 2743 tablet_bootstrap.cc:492] T f9057d80b6a64f01937179121f461a1f P af01fe7dfaeb464392cfef10732beb0d: Bootstrap starting.
I20260928 05:17:20.083500 2744 tablet_bootstrap.cc:492] T f9057d80b6a64f01937179121f461a1f P 64aee4950dd845c6b9174d260e230a36: Bootstrap starting.
I20260928 05:17:20.084007 2745 tablet_bootstrap.cc:492] T f9057d80b6a64f01937179121f461a1f P 3de53bd0c5ee445fbfbcb66500285846: Bootstrap starting.
I20260928 05:17:20.084087 2743 tablet_bootstrap.cc:654] T f9057d80b6a64f01937179121f461a1f P af01fe7dfaeb464392cfef10732beb0d: Neither blocks nor log segments found. Creating new log.
I20260928 05:17:20.085562 2745 tablet_bootstrap.cc:654] T f9057d80b6a64f01937179121f461a1f P 3de53bd0c5ee445fbfbcb66500285846: Neither blocks nor log segments found. Creating new log.
I20260928 05:17:20.086628 2744 tablet_bootstrap.cc:654] T f9057d80b6a64f01937179121f461a1f P 64aee4950dd845c6b9174d260e230a36: Neither blocks nor log segments found. Creating new log.
I20260928 05:17:20.087809 2743 tablet_bootstrap.cc:492] T f9057d80b6a64f01937179121f461a1f P af01fe7dfaeb464392cfef10732beb0d: No bootstrap required, opened a new log
I20260928 05:17:20.087936 2743 ts_tablet_manager.cc:1397] T f9057d80b6a64f01937179121f461a1f P af01fe7dfaeb464392cfef10732beb0d: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260928 05:17:20.088335 2745 tablet_bootstrap.cc:492] T f9057d80b6a64f01937179121f461a1f P 3de53bd0c5ee445fbfbcb66500285846: No bootstrap required, opened a new log
I20260928 05:17:20.088335 2744 tablet_bootstrap.cc:492] T f9057d80b6a64f01937179121f461a1f P 64aee4950dd845c6b9174d260e230a36: No bootstrap required, opened a new log
I20260928 05:17:20.088462 2745 ts_tablet_manager.cc:1397] T f9057d80b6a64f01937179121f461a1f P 3de53bd0c5ee445fbfbcb66500285846: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260928 05:17:20.088462 2744 ts_tablet_manager.cc:1397] T f9057d80b6a64f01937179121f461a1f P 64aee4950dd845c6b9174d260e230a36: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260928 05:17:20.088481 2743 raft_consensus.cc:348] T f9057d80b6a64f01937179121f461a1f P af01fe7dfaeb464392cfef10732beb0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64aee4950dd845c6b9174d260e230a36" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43645 } } peers { permanent_uuid: "af01fe7dfaeb464392cfef10732beb0d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37821 } } peers { permanent_uuid: "3de53bd0c5ee445fbfbcb66500285846" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43981 } }
I20260928 05:17:20.088685 2743 raft_consensus.cc:374] T f9057d80b6a64f01937179121f461a1f P af01fe7dfaeb464392cfef10732beb0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:17:20.088788 2743 raft_consensus.cc:729] T f9057d80b6a64f01937179121f461a1f P af01fe7dfaeb464392cfef10732beb0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af01fe7dfaeb464392cfef10732beb0d, State: Initialized, Role: FOLLOWER
I20260928 05:17:20.089030 2743 consensus_queue.cc:260] T f9057d80b6a64f01937179121f461a1f P af01fe7dfaeb464392cfef10732beb0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64aee4950dd845c6b9174d260e230a36" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43645 } } peers { permanent_uuid: "af01fe7dfaeb464392cfef10732beb0d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37821 } } peers { permanent_uuid: "3de53bd0c5ee445fbfbcb66500285846" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43981 } }
I20260928 05:17:20.089234 2744 raft_consensus.cc:348] T f9057d80b6a64f01937179121f461a1f P 64aee4950dd845c6b9174d260e230a36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64aee4950dd845c6b9174d260e230a36" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43645 } } peers { permanent_uuid: "af01fe7dfaeb464392cfef10732beb0d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37821 } } peers { permanent_uuid: "3de53bd0c5ee445fbfbcb66500285846" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43981 } }
I20260928 05:17:20.089234 2745 raft_consensus.cc:348] T f9057d80b6a64f01937179121f461a1f P 3de53bd0c5ee445fbfbcb66500285846 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64aee4950dd845c6b9174d260e230a36" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43645 } } peers { permanent_uuid: "af01fe7dfaeb464392cfef10732beb0d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37821 } } peers { permanent_uuid: "3de53bd0c5ee445fbfbcb66500285846" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43981 } }
I20260928 05:17:20.089439 2744 raft_consensus.cc:374] T f9057d80b6a64f01937179121f461a1f P 64aee4950dd845c6b9174d260e230a36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:17:20.089495 2745 raft_consensus.cc:374] T f9057d80b6a64f01937179121f461a1f P 3de53bd0c5ee445fbfbcb66500285846 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:17:20.089521 2744 raft_consensus.cc:729] T f9057d80b6a64f01937179121f461a1f P 64aee4950dd845c6b9174d260e230a36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64aee4950dd845c6b9174d260e230a36, State: Initialized, Role: FOLLOWER
I20260928 05:17:20.089569 2745 raft_consensus.cc:729] T f9057d80b6a64f01937179121f461a1f P 3de53bd0c5ee445fbfbcb66500285846 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3de53bd0c5ee445fbfbcb66500285846, State: Initialized, Role: FOLLOWER
I20260928 05:17:20.089771 2745 consensus_queue.cc:260] T f9057d80b6a64f01937179121f461a1f P 3de53bd0c5ee445fbfbcb66500285846 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64aee4950dd845c6b9174d260e230a36" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43645 } } peers { permanent_uuid: "af01fe7dfaeb464392cfef10732beb0d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37821 } } peers { permanent_uuid: "3de53bd0c5ee445fbfbcb66500285846" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43981 } }
I20260928 05:17:20.089771 2744 consensus_queue.cc:260] T f9057d80b6a64f01937179121f461a1f P 64aee4950dd845c6b9174d260e230a36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64aee4950dd845c6b9174d260e230a36" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43645 } } peers { permanent_uuid: "af01fe7dfaeb464392cfef10732beb0d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37821 } } peers { permanent_uuid: "3de53bd0c5ee445fbfbcb66500285846" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43981 } }
I20260928 05:17:20.090292 2745 ts_tablet_manager.cc:1428] T f9057d80b6a64f01937179121f461a1f P 3de53bd0c5ee445fbfbcb66500285846: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260928 05:17:20.090343 2744 ts_tablet_manager.cc:1428] T f9057d80b6a64f01937179121f461a1f P 64aee4950dd845c6b9174d260e230a36: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 05:17:20.097292 2743 ts_tablet_manager.cc:1428] T f9057d80b6a64f01937179121f461a1f P af01fe7dfaeb464392cfef10732beb0d: Time spent starting tablet: real 0.009s user 0.002s sys 0.007s
I20260928 05:17:20.204208 2725 raft_consensus.cc:482] T f9057d80b6a64f01937179121f461a1f P 3de53bd0c5ee445fbfbcb66500285846 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:17:20.204398 2725 raft_consensus.cc:504] T f9057d80b6a64f01937179121f461a1f P 3de53bd0c5ee445fbfbcb66500285846 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64aee4950dd845c6b9174d260e230a36" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43645 } } peers { permanent_uuid: "af01fe7dfaeb464392cfef10732beb0d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37821 } } peers { permanent_uuid: "3de53bd0c5ee445fbfbcb66500285846" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43981 } }
I20260928 05:17:20.205096 2725 leader_election.cc:288] T f9057d80b6a64f01937179121f461a1f P 3de53bd0c5ee445fbfbcb66500285846 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64aee4950dd845c6b9174d260e230a36 (127.0.97.1:43645), af01fe7dfaeb464392cfef10732beb0d (127.0.97.2:37821)
I20260928 05:17:20.205631 2426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9057d80b6a64f01937179121f461a1f" candidate_uuid: "3de53bd0c5ee445fbfbcb66500285846" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af01fe7dfaeb464392cfef10732beb0d" is_pre_election: true
I20260928 05:17:20.205626 2286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9057d80b6a64f01937179121f461a1f" candidate_uuid: "3de53bd0c5ee445fbfbcb66500285846" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64aee4950dd845c6b9174d260e230a36" is_pre_election: true
I20260928 05:17:20.205861 2286 raft_consensus.cc:2454] T f9057d80b6a64f01937179121f461a1f P 64aee4950dd845c6b9174d260e230a36 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3de53bd0c5ee445fbfbcb66500285846 in term 0.
I20260928 05:17:20.205859 2426 raft_consensus.cc:2454] T f9057d80b6a64f01937179121f461a1f P af01fe7dfaeb464392cfef10732beb0d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3de53bd0c5ee445fbfbcb66500285846 in term 0.
I20260928 05:17:20.206290 2496 leader_election.cc:302] T f9057d80b6a64f01937179121f461a1f P 3de53bd0c5ee445fbfbcb66500285846 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3de53bd0c5ee445fbfbcb66500285846, 64aee4950dd845c6b9174d260e230a36, af01fe7dfaeb464392cfef10732beb0d; no voters:
I20260928 05:17:20.206542 2725 raft_consensus.cc:2781] T f9057d80b6a64f01937179121f461a1f P 3de53bd0c5ee445fbfbcb66500285846 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 05:17:20.206646 2725 raft_consensus.cc:482] T f9057d80b6a64f01937179121f461a1f P 3de53bd0c5ee445fbfbcb66500285846 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 05:17:20.206727 2725 raft_consensus.cc:3037] T f9057d80b6a64f01937179121f461a1f P 3de53bd0c5ee445fbfbcb66500285846 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:17:20.208689 2725 raft_consensus.cc:504] T f9057d80b6a64f01937179121f461a1f P 3de53bd0c5ee445fbfbcb66500285846 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64aee4950dd845c6b9174d260e230a36" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43645 } } peers { permanent_uuid: "af01fe7dfaeb464392cfef10732beb0d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37821 } } peers { permanent_uuid: "3de53bd0c5ee445fbfbcb66500285846" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43981 } }
I20260928 05:17:20.209218 2725 leader_election.cc:288] T f9057d80b6a64f01937179121f461a1f P 3de53bd0c5ee445fbfbcb66500285846 [CANDIDATE]: Term 1 election: Requested vote from peers 64aee4950dd845c6b9174d260e230a36 (127.0.97.1:43645), af01fe7dfaeb464392cfef10732beb0d (127.0.97.2:37821)
I20260928 05:17:20.209609 2286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9057d80b6a64f01937179121f461a1f" candidate_uuid: "3de53bd0c5ee445fbfbcb66500285846" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64aee4950dd845c6b9174d260e230a36"
I20260928 05:17:20.209669 2426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9057d80b6a64f01937179121f461a1f" candidate_uuid: "3de53bd0c5ee445fbfbcb66500285846" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af01fe7dfaeb464392cfef10732beb0d"
I20260928 05:17:20.209760 2286 raft_consensus.cc:3037] T f9057d80b6a64f01937179121f461a1f P 64aee4950dd845c6b9174d260e230a36 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:17:20.209843 2426 raft_consensus.cc:3037] T f9057d80b6a64f01937179121f461a1f P af01fe7dfaeb464392cfef10732beb0d [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:17:20.211623 2426 raft_consensus.cc:2454] T f9057d80b6a64f01937179121f461a1f P af01fe7dfaeb464392cfef10732beb0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3de53bd0c5ee445fbfbcb66500285846 in term 1.
I20260928 05:17:20.211627 2286 raft_consensus.cc:2454] T f9057d80b6a64f01937179121f461a1f P 64aee4950dd845c6b9174d260e230a36 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3de53bd0c5ee445fbfbcb66500285846 in term 1.
I20260928 05:17:20.211957 2496 leader_election.cc:302] T f9057d80b6a64f01937179121f461a1f P 3de53bd0c5ee445fbfbcb66500285846 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3de53bd0c5ee445fbfbcb66500285846, 64aee4950dd845c6b9174d260e230a36; no voters:
I20260928 05:17:20.212194 2725 raft_consensus.cc:2781] T f9057d80b6a64f01937179121f461a1f P 3de53bd0c5ee445fbfbcb66500285846 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:17:20.212328 2725 raft_consensus.cc:686] T f9057d80b6a64f01937179121f461a1f P 3de53bd0c5ee445fbfbcb66500285846 [term 1 LEADER]: Becoming Leader. State: Replica: 3de53bd0c5ee445fbfbcb66500285846, State: Running, Role: LEADER
I20260928 05:17:20.212563 2725 consensus_queue.cc:237] T f9057d80b6a64f01937179121f461a1f P 3de53bd0c5ee445fbfbcb66500285846 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64aee4950dd845c6b9174d260e230a36" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43645 } } peers { permanent_uuid: "af01fe7dfaeb464392cfef10732beb0d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37821 } } peers { permanent_uuid: "3de53bd0c5ee445fbfbcb66500285846" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43981 } }
I20260928 05:17:20.215046 2131 catalog_manager.cc:5495] T f9057d80b6a64f01937179121f461a1f P 3de53bd0c5ee445fbfbcb66500285846 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3de53bd0c5ee445fbfbcb66500285846 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "3de53bd0c5ee445fbfbcb66500285846" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64aee4950dd845c6b9174d260e230a36" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af01fe7dfaeb464392cfef10732beb0d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3de53bd0c5ee445fbfbcb66500285846" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43981 } health_report { overall_health: HEALTHY } } }
I20260928 05:17:20.241501 2752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:20.236758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37900 (local address 127.0.97.3:43981)
0928 05:17:20.237675 (+ 917us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:20.237679 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:20.237696 (+ 17us) server_negotiation.cc:407] Connection header received
0928 05:17:20.237741 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:20.237744 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:20.237799 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:20.237877 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:17:20.238150 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:20.238839 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:20.240022 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:20.240423 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:20.240510 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:20.240868 (+ 358us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:17:20.240988 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:17:20.241051 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:20.241150 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 05:17:20.241206 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":813,"thread_start_us":291,"threads_started":1}
I20260928 05:17:20.248927 2426 raft_consensus.cc:1261] T f9057d80b6a64f01937179121f461a1f P af01fe7dfaeb464392cfef10732beb0d [term 1 FOLLOWER]: Refusing update from remote peer 3de53bd0c5ee445fbfbcb66500285846: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:17:20.249168 2286 raft_consensus.cc:1261] T f9057d80b6a64f01937179121f461a1f P 64aee4950dd845c6b9174d260e230a36 [term 1 FOLLOWER]: Refusing update from remote peer 3de53bd0c5ee445fbfbcb66500285846: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:17:20.249399 2722 consensus_queue.cc:1035] T f9057d80b6a64f01937179121f461a1f P 3de53bd0c5ee445fbfbcb66500285846 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af01fe7dfaeb464392cfef10732beb0d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:17:20.249711 2725 consensus_queue.cc:1035] T f9057d80b6a64f01937179121f461a1f P 3de53bd0c5ee445fbfbcb66500285846 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64aee4950dd845c6b9174d260e230a36" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:17:20.290132 2130 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59602:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:17:20.290354 2130 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59602:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:17:20.292953 2130 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aad10d5c9dbc434e9742c4102ad69acd: Sending DeleteTablet for 3 replicas of tablet f9057d80b6a64f01937179121f461a1f
I20260928 05:17:20.293658 2267 tablet_service.cc:1514] Processing DeleteTablet for tablet f9057d80b6a64f01937179121f461a1f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:17:20 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57296
I20260928 05:17:20.293905 2733 tablet_replica.cc:330] T f9057d80b6a64f01937179121f461a1f P 64aee4950dd845c6b9174d260e230a36: stopping tablet replica
I20260928 05:17:20.293996 2405 tablet_service.cc:1514] Processing DeleteTablet for tablet f9057d80b6a64f01937179121f461a1f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:17:20 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46298
I20260928 05:17:20.294093 2733 raft_consensus.cc:2229] T f9057d80b6a64f01937179121f461a1f P 64aee4950dd845c6b9174d260e230a36 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:17:20.294222 2734 tablet_replica.cc:330] T f9057d80b6a64f01937179121f461a1f P af01fe7dfaeb464392cfef10732beb0d: stopping tablet replica
I20260928 05:17:20.294245 2733 raft_consensus.cc:2258] T f9057d80b6a64f01937179121f461a1f P 64aee4950dd845c6b9174d260e230a36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:17:20.294389 2734 raft_consensus.cc:2229] T f9057d80b6a64f01937179121f461a1f P af01fe7dfaeb464392cfef10732beb0d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:17:20.294488 2545 tablet_service.cc:1514] Processing DeleteTablet for tablet f9057d80b6a64f01937179121f461a1f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:17:20 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37880
I20260928 05:17:20.294548 2734 raft_consensus.cc:2258] T f9057d80b6a64f01937179121f461a1f P af01fe7dfaeb464392cfef10732beb0d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:17:20.294746 2732 tablet_replica.cc:330] T f9057d80b6a64f01937179121f461a1f P 3de53bd0c5ee445fbfbcb66500285846: stopping tablet replica
I20260928 05:17:20.294920 2732 raft_consensus.cc:2229] T f9057d80b6a64f01937179121f461a1f P 3de53bd0c5ee445fbfbcb66500285846 [term 1 LEADER]: Raft consensus shutting down.
I20260928 05:17:20.295202 2732 raft_consensus.cc:2258] T f9057d80b6a64f01937179121f461a1f P 3de53bd0c5ee445fbfbcb66500285846 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:17:20.295408 2733 ts_tablet_manager.cc:1905] T f9057d80b6a64f01937179121f461a1f P 64aee4950dd845c6b9174d260e230a36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:17:20.295624 2734 ts_tablet_manager.cc:1905] T f9057d80b6a64f01937179121f461a1f P af01fe7dfaeb464392cfef10732beb0d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:17:20.296281 2732 ts_tablet_manager.cc:1905] T f9057d80b6a64f01937179121f461a1f P 3de53bd0c5ee445fbfbcb66500285846: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:17:20.301527 2734 ts_tablet_manager.cc:1918] T f9057d80b6a64f01937179121f461a1f P af01fe7dfaeb464392cfef10732beb0d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 05:17:20.301546 2733 ts_tablet_manager.cc:1918] T f9057d80b6a64f01937179121f461a1f P 64aee4950dd845c6b9174d260e230a36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 05:17:20.301662 2733 log.cc:1196] T f9057d80b6a64f01937179121f461a1f P 64aee4950dd845c6b9174d260e230a36: Deleting WAL directory at /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/minicluster-data/ts-0/wal/wals/f9057d80b6a64f01937179121f461a1f
I20260928 05:17:20.302089 2733 ts_tablet_manager.cc:1939] T f9057d80b6a64f01937179121f461a1f P 64aee4950dd845c6b9174d260e230a36: Deleting consensus metadata
I20260928 05:17:20.302358 2732 ts_tablet_manager.cc:1918] T f9057d80b6a64f01937179121f461a1f P 3de53bd0c5ee445fbfbcb66500285846: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 05:17:20.302510 2732 log.cc:1196] T f9057d80b6a64f01937179121f461a1f P 3de53bd0c5ee445fbfbcb66500285846: Deleting WAL directory at /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/minicluster-data/ts-2/wal/wals/f9057d80b6a64f01937179121f461a1f
I20260928 05:17:20.302881 2732 ts_tablet_manager.cc:1939] T f9057d80b6a64f01937179121f461a1f P 3de53bd0c5ee445fbfbcb66500285846: Deleting consensus metadata
I20260928 05:17:20.302882 2130 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59602:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 05:17:20.303157 2115 catalog_manager.cc:4841] TS 64aee4950dd845c6b9174d260e230a36 (127.0.97.1:43645): tablet f9057d80b6a64f01937179121f461a1f (table test-table [id=0061caf983a14ac7aaba65d43c654e82]) successfully deleted
W20260928 05:17:20.303474 2130 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 05:17:20.303762 2117 catalog_manager.cc:4841] TS 3de53bd0c5ee445fbfbcb66500285846 (127.0.97.3:43981): tablet f9057d80b6a64f01937179121f461a1f (table test-table [id=0061caf983a14ac7aaba65d43c654e82]) successfully deleted
I20260928 05:17:20.304453 2734 log.cc:1196] T f9057d80b6a64f01937179121f461a1f P af01fe7dfaeb464392cfef10732beb0d: Deleting WAL directory at /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/minicluster-data/ts-1/wal/wals/f9057d80b6a64f01937179121f461a1f
I20260928 05:17:20.304841 2734 ts_tablet_manager.cc:1939] T f9057d80b6a64f01937179121f461a1f P af01fe7dfaeb464392cfef10732beb0d: Deleting consensus metadata
I20260928 05:17:20.305673 2116 catalog_manager.cc:4841] TS af01fe7dfaeb464392cfef10732beb0d (127.0.97.2:37821): tablet f9057d80b6a64f01937179121f461a1f (table test-table [id=0061caf983a14ac7aaba65d43c654e82]) successfully deleted
I20260928 05:17:20.309554 2267 tablet_service.cc:1467] Processing CreateTablet for tablet 881106352e1a42469017cb679ef4fa3e (DEFAULT_TABLE table=test-table [id=49e7ee3690724c7e8d9c2f64deb94bd2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:17:20.309923 2267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 881106352e1a42469017cb679ef4fa3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:17:20.309911 2545 tablet_service.cc:1467] Processing CreateTablet for tablet 881106352e1a42469017cb679ef4fa3e (DEFAULT_TABLE table=test-table [id=49e7ee3690724c7e8d9c2f64deb94bd2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:17:20.310256 2545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 881106352e1a42469017cb679ef4fa3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:17:20.314081 2744 tablet_bootstrap.cc:492] T 881106352e1a42469017cb679ef4fa3e P 64aee4950dd845c6b9174d260e230a36: Bootstrap starting.
I20260928 05:17:20.315416 2744 tablet_bootstrap.cc:654] T 881106352e1a42469017cb679ef4fa3e P 64aee4950dd845c6b9174d260e230a36: Neither blocks nor log segments found. Creating new log.
I20260928 05:17:20.316387 2745 tablet_bootstrap.cc:492] T 881106352e1a42469017cb679ef4fa3e P 3de53bd0c5ee445fbfbcb66500285846: Bootstrap starting.
I20260928 05:17:20.317054 2405 tablet_service.cc:1467] Processing CreateTablet for tablet 881106352e1a42469017cb679ef4fa3e (DEFAULT_TABLE table=test-table [id=49e7ee3690724c7e8d9c2f64deb94bd2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:17:20.317217 2744 tablet_bootstrap.cc:492] T 881106352e1a42469017cb679ef4fa3e P 64aee4950dd845c6b9174d260e230a36: No bootstrap required, opened a new log
I20260928 05:17:20.317363 2744 ts_tablet_manager.cc:1397] T 881106352e1a42469017cb679ef4fa3e P 64aee4950dd845c6b9174d260e230a36: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 05:17:20.317394 2405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 881106352e1a42469017cb679ef4fa3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:17:20.317970 2745 tablet_bootstrap.cc:654] T 881106352e1a42469017cb679ef4fa3e P 3de53bd0c5ee445fbfbcb66500285846: Neither blocks nor log segments found. Creating new log.
I20260928 05:17:20.318583 2744 raft_consensus.cc:348] T 881106352e1a42469017cb679ef4fa3e P 64aee4950dd845c6b9174d260e230a36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64aee4950dd845c6b9174d260e230a36" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43645 } } peers { permanent_uuid: "af01fe7dfaeb464392cfef10732beb0d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37821 } } peers { permanent_uuid: "3de53bd0c5ee445fbfbcb66500285846" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43981 } }
I20260928 05:17:20.318827 2744 raft_consensus.cc:374] T 881106352e1a42469017cb679ef4fa3e P 64aee4950dd845c6b9174d260e230a36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:17:20.318910 2744 raft_consensus.cc:729] T 881106352e1a42469017cb679ef4fa3e P 64aee4950dd845c6b9174d260e230a36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64aee4950dd845c6b9174d260e230a36, State: Initialized, Role: FOLLOWER
I20260928 05:17:20.319124 2744 consensus_queue.cc:260] T 881106352e1a42469017cb679ef4fa3e P 64aee4950dd845c6b9174d260e230a36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64aee4950dd845c6b9174d260e230a36" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43645 } } peers { permanent_uuid: "af01fe7dfaeb464392cfef10732beb0d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37821 } } peers { permanent_uuid: "3de53bd0c5ee445fbfbcb66500285846" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43981 } }
I20260928 05:17:20.319633 2744 ts_tablet_manager.cc:1428] T 881106352e1a42469017cb679ef4fa3e P 64aee4950dd845c6b9174d260e230a36: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260928 05:17:20.320015 2745 tablet_bootstrap.cc:492] T 881106352e1a42469017cb679ef4fa3e P 3de53bd0c5ee445fbfbcb66500285846: No bootstrap required, opened a new log
I20260928 05:17:20.320156 2745 ts_tablet_manager.cc:1397] T 881106352e1a42469017cb679ef4fa3e P 3de53bd0c5ee445fbfbcb66500285846: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260928 05:17:20.320613 2745 raft_consensus.cc:348] T 881106352e1a42469017cb679ef4fa3e P 3de53bd0c5ee445fbfbcb66500285846 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64aee4950dd845c6b9174d260e230a36" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43645 } } peers { permanent_uuid: "af01fe7dfaeb464392cfef10732beb0d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37821 } } peers { permanent_uuid: "3de53bd0c5ee445fbfbcb66500285846" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43981 } }
I20260928 05:17:20.320756 2745 raft_consensus.cc:374] T 881106352e1a42469017cb679ef4fa3e P 3de53bd0c5ee445fbfbcb66500285846 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:17:20.320799 2745 raft_consensus.cc:729] T 881106352e1a42469017cb679ef4fa3e P 3de53bd0c5ee445fbfbcb66500285846 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3de53bd0c5ee445fbfbcb66500285846, State: Initialized, Role: FOLLOWER
I20260928 05:17:20.321053 2745 consensus_queue.cc:260] T 881106352e1a42469017cb679ef4fa3e P 3de53bd0c5ee445fbfbcb66500285846 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64aee4950dd845c6b9174d260e230a36" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43645 } } peers { permanent_uuid: "af01fe7dfaeb464392cfef10732beb0d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37821 } } peers { permanent_uuid: "3de53bd0c5ee445fbfbcb66500285846" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43981 } }
I20260928 05:17:20.321252 2757 raft_consensus.cc:482] T 881106352e1a42469017cb679ef4fa3e P 3de53bd0c5ee445fbfbcb66500285846 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:17:20.321419 2757 raft_consensus.cc:504] T 881106352e1a42469017cb679ef4fa3e P 3de53bd0c5ee445fbfbcb66500285846 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64aee4950dd845c6b9174d260e230a36" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43645 } } peers { permanent_uuid: "af01fe7dfaeb464392cfef10732beb0d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37821 } } peers { permanent_uuid: "3de53bd0c5ee445fbfbcb66500285846" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43981 } }
I20260928 05:17:20.322117 2757 leader_election.cc:288] T 881106352e1a42469017cb679ef4fa3e P 3de53bd0c5ee445fbfbcb66500285846 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64aee4950dd845c6b9174d260e230a36 (127.0.97.1:43645), af01fe7dfaeb464392cfef10732beb0d (127.0.97.2:37821)
I20260928 05:17:20.322813 2425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "881106352e1a42469017cb679ef4fa3e" candidate_uuid: "3de53bd0c5ee445fbfbcb66500285846" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af01fe7dfaeb464392cfef10732beb0d" is_pre_election: true
I20260928 05:17:20.323180 2745 ts_tablet_manager.cc:1428] T 881106352e1a42469017cb679ef4fa3e P 3de53bd0c5ee445fbfbcb66500285846: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
W20260928 05:17:20.323514 2497 leader_election.cc:341] T 881106352e1a42469017cb679ef4fa3e P 3de53bd0c5ee445fbfbcb66500285846 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af01fe7dfaeb464392cfef10732beb0d (127.0.97.2:37821): Illegal state: must be running to vote when last-logged opid is not known
I20260928 05:17:20.324010 2743 tablet_bootstrap.cc:492] T 881106352e1a42469017cb679ef4fa3e P af01fe7dfaeb464392cfef10732beb0d: Bootstrap starting.
I20260928 05:17:20.324080 2286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "881106352e1a42469017cb679ef4fa3e" candidate_uuid: "3de53bd0c5ee445fbfbcb66500285846" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64aee4950dd845c6b9174d260e230a36" is_pre_election: true
I20260928 05:17:20.324267 2286 raft_consensus.cc:2454] T 881106352e1a42469017cb679ef4fa3e P 64aee4950dd845c6b9174d260e230a36 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3de53bd0c5ee445fbfbcb66500285846 in term 0.
I20260928 05:17:20.324738 2496 leader_election.cc:302] T 881106352e1a42469017cb679ef4fa3e P 3de53bd0c5ee445fbfbcb66500285846 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3de53bd0c5ee445fbfbcb66500285846, 64aee4950dd845c6b9174d260e230a36; no voters: af01fe7dfaeb464392cfef10732beb0d
I20260928 05:17:20.325014 2757 raft_consensus.cc:2781] T 881106352e1a42469017cb679ef4fa3e P 3de53bd0c5ee445fbfbcb66500285846 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 05:17:20.325124 2757 raft_consensus.cc:482] T 881106352e1a42469017cb679ef4fa3e P 3de53bd0c5ee445fbfbcb66500285846 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 05:17:20.325209 2757 raft_consensus.cc:3037] T 881106352e1a42469017cb679ef4fa3e P 3de53bd0c5ee445fbfbcb66500285846 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:17:20.325335 2743 tablet_bootstrap.cc:654] T 881106352e1a42469017cb679ef4fa3e P af01fe7dfaeb464392cfef10732beb0d: Neither blocks nor log segments found. Creating new log.
I20260928 05:17:20.326905 2743 tablet_bootstrap.cc:492] T 881106352e1a42469017cb679ef4fa3e P af01fe7dfaeb464392cfef10732beb0d: No bootstrap required, opened a new log
I20260928 05:17:20.327039 2743 ts_tablet_manager.cc:1397] T 881106352e1a42469017cb679ef4fa3e P af01fe7dfaeb464392cfef10732beb0d: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260928 05:17:20.327616 2743 raft_consensus.cc:348] T 881106352e1a42469017cb679ef4fa3e P af01fe7dfaeb464392cfef10732beb0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64aee4950dd845c6b9174d260e230a36" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43645 } } peers { permanent_uuid: "af01fe7dfaeb464392cfef10732beb0d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37821 } } peers { permanent_uuid: "3de53bd0c5ee445fbfbcb66500285846" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43981 } }
I20260928 05:17:20.327786 2743 raft_consensus.cc:374] T 881106352e1a42469017cb679ef4fa3e P af01fe7dfaeb464392cfef10732beb0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:17:20.327720 2757 raft_consensus.cc:504] T 881106352e1a42469017cb679ef4fa3e P 3de53bd0c5ee445fbfbcb66500285846 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64aee4950dd845c6b9174d260e230a36" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43645 } } peers { permanent_uuid: "af01fe7dfaeb464392cfef10732beb0d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37821 } } peers { permanent_uuid: "3de53bd0c5ee445fbfbcb66500285846" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43981 } }
I20260928 05:17:20.327855 2743 raft_consensus.cc:729] T 881106352e1a42469017cb679ef4fa3e P af01fe7dfaeb464392cfef10732beb0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af01fe7dfaeb464392cfef10732beb0d, State: Initialized, Role: FOLLOWER
I20260928 05:17:20.328058 2743 consensus_queue.cc:260] T 881106352e1a42469017cb679ef4fa3e P af01fe7dfaeb464392cfef10732beb0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64aee4950dd845c6b9174d260e230a36" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43645 } } peers { permanent_uuid: "af01fe7dfaeb464392cfef10732beb0d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37821 } } peers { permanent_uuid: "3de53bd0c5ee445fbfbcb66500285846" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43981 } }
I20260928 05:17:20.328176 2757 leader_election.cc:288] T 881106352e1a42469017cb679ef4fa3e P 3de53bd0c5ee445fbfbcb66500285846 [CANDIDATE]: Term 1 election: Requested vote from peers 64aee4950dd845c6b9174d260e230a36 (127.0.97.1:43645), af01fe7dfaeb464392cfef10732beb0d (127.0.97.2:37821)
I20260928 05:17:20.328610 2743 ts_tablet_manager.cc:1428] T 881106352e1a42469017cb679ef4fa3e P af01fe7dfaeb464392cfef10732beb0d: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260928 05:17:20.328884 2425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "881106352e1a42469017cb679ef4fa3e" candidate_uuid: "3de53bd0c5ee445fbfbcb66500285846" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af01fe7dfaeb464392cfef10732beb0d"
I20260928 05:17:20.329068 2425 raft_consensus.cc:3037] T 881106352e1a42469017cb679ef4fa3e P af01fe7dfaeb464392cfef10732beb0d [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:17:20.331058 2425 raft_consensus.cc:2454] T 881106352e1a42469017cb679ef4fa3e P af01fe7dfaeb464392cfef10732beb0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3de53bd0c5ee445fbfbcb66500285846 in term 1.
I20260928 05:17:20.331404 2497 leader_election.cc:302] T 881106352e1a42469017cb679ef4fa3e P 3de53bd0c5ee445fbfbcb66500285846 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3de53bd0c5ee445fbfbcb66500285846, af01fe7dfaeb464392cfef10732beb0d; no voters:
I20260928 05:17:20.331537 2286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "881106352e1a42469017cb679ef4fa3e" candidate_uuid: "3de53bd0c5ee445fbfbcb66500285846" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64aee4950dd845c6b9174d260e230a36"
I20260928 05:17:20.331636 2757 raft_consensus.cc:2781] T 881106352e1a42469017cb679ef4fa3e P 3de53bd0c5ee445fbfbcb66500285846 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:17:20.331736 2286 raft_consensus.cc:3037] T 881106352e1a42469017cb679ef4fa3e P 64aee4950dd845c6b9174d260e230a36 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:17:20.331766 2757 raft_consensus.cc:686] T 881106352e1a42469017cb679ef4fa3e P 3de53bd0c5ee445fbfbcb66500285846 [term 1 LEADER]: Becoming Leader. State: Replica: 3de53bd0c5ee445fbfbcb66500285846, State: Running, Role: LEADER
I20260928 05:17:20.331995 2757 consensus_queue.cc:237] T 881106352e1a42469017cb679ef4fa3e P 3de53bd0c5ee445fbfbcb66500285846 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64aee4950dd845c6b9174d260e230a36" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43645 } } peers { permanent_uuid: "af01fe7dfaeb464392cfef10732beb0d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37821 } } peers { permanent_uuid: "3de53bd0c5ee445fbfbcb66500285846" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43981 } }
I20260928 05:17:20.334326 2286 raft_consensus.cc:2454] T 881106352e1a42469017cb679ef4fa3e P 64aee4950dd845c6b9174d260e230a36 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3de53bd0c5ee445fbfbcb66500285846 in term 1.
I20260928 05:17:20.334329 2129 catalog_manager.cc:5495] T 881106352e1a42469017cb679ef4fa3e P 3de53bd0c5ee445fbfbcb66500285846 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3de53bd0c5ee445fbfbcb66500285846 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "3de53bd0c5ee445fbfbcb66500285846" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64aee4950dd845c6b9174d260e230a36" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af01fe7dfaeb464392cfef10732beb0d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3de53bd0c5ee445fbfbcb66500285846" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43981 } health_report { overall_health: HEALTHY } } }
I20260928 05:17:20.352321 2286 raft_consensus.cc:1261] T 881106352e1a42469017cb679ef4fa3e P 64aee4950dd845c6b9174d260e230a36 [term 1 FOLLOWER]: Refusing update from remote peer 3de53bd0c5ee445fbfbcb66500285846: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:17:20.352563 2425 raft_consensus.cc:1261] T 881106352e1a42469017cb679ef4fa3e P af01fe7dfaeb464392cfef10732beb0d [term 1 FOLLOWER]: Refusing update from remote peer 3de53bd0c5ee445fbfbcb66500285846: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:17:20.352790 2640 consensus_queue.cc:1035] T 881106352e1a42469017cb679ef4fa3e P 3de53bd0c5ee445fbfbcb66500285846 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64aee4950dd845c6b9174d260e230a36" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:17:20.353005 2757 consensus_queue.cc:1035] T 881106352e1a42469017cb679ef4fa3e P 3de53bd0c5ee445fbfbcb66500285846 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af01fe7dfaeb464392cfef10732beb0d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260928 05:17:20.369110 2714 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 05:17:21.946343 2131 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59602:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:17:21.946530 2131 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59602:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:17:21.950344 2131 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aad10d5c9dbc434e9742c4102ad69acd: Sending DeleteTablet for 3 replicas of tablet 881106352e1a42469017cb679ef4fa3e
I20260928 05:17:21.950914 2267 tablet_service.cc:1514] Processing DeleteTablet for tablet 881106352e1a42469017cb679ef4fa3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:17:21 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57296
I20260928 05:17:21.951004 2405 tablet_service.cc:1514] Processing DeleteTablet for tablet 881106352e1a42469017cb679ef4fa3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:17:21 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46298
I20260928 05:17:21.951092 2545 tablet_service.cc:1514] Processing DeleteTablet for tablet 881106352e1a42469017cb679ef4fa3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:17:21 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37880
I20260928 05:17:21.951545 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 2202
I20260928 05:17:21.983738 2786 tablet_replica.cc:330] T 881106352e1a42469017cb679ef4fa3e P 64aee4950dd845c6b9174d260e230a36: stopping tablet replica
I20260928 05:17:21.983997 2786 raft_consensus.cc:2229] T 881106352e1a42469017cb679ef4fa3e P 64aee4950dd845c6b9174d260e230a36 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:17:21.984248 2786 raft_consensus.cc:2258] T 881106352e1a42469017cb679ef4fa3e P 64aee4950dd845c6b9174d260e230a36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:17:21.985773 2786 ts_tablet_manager.cc:1905] T 881106352e1a42469017cb679ef4fa3e P 64aee4950dd845c6b9174d260e230a36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:17:21.989001 2788 tablet_replica.cc:330] T 881106352e1a42469017cb679ef4fa3e P 3de53bd0c5ee445fbfbcb66500285846: stopping tablet replica
I20260928 05:17:21.989236 2788 raft_consensus.cc:2229] T 881106352e1a42469017cb679ef4fa3e P 3de53bd0c5ee445fbfbcb66500285846 [term 1 LEADER]: Raft consensus shutting down.
I20260928 05:17:21.989262 2787 tablet_replica.cc:330] T 881106352e1a42469017cb679ef4fa3e P af01fe7dfaeb464392cfef10732beb0d: stopping tablet replica
I20260928 05:17:21.989434 2787 raft_consensus.cc:2229] T 881106352e1a42469017cb679ef4fa3e P af01fe7dfaeb464392cfef10732beb0d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:17:21.989540 2788 raft_consensus.cc:2258] T 881106352e1a42469017cb679ef4fa3e P 3de53bd0c5ee445fbfbcb66500285846 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:17:21.989686 2787 raft_consensus.cc:2258] T 881106352e1a42469017cb679ef4fa3e P af01fe7dfaeb464392cfef10732beb0d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:17:21.990988 2788 ts_tablet_manager.cc:1905] T 881106352e1a42469017cb679ef4fa3e P 3de53bd0c5ee445fbfbcb66500285846: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 28 05:17:21 dist-test-slave-whsn krb5kdc[2086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572640, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260928 05:17:21.992209 2787 ts_tablet_manager.cc:1905] T 881106352e1a42469017cb679ef4fa3e P af01fe7dfaeb464392cfef10732beb0d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:17:21.992332 2786 ts_tablet_manager.cc:1918] T 881106352e1a42469017cb679ef4fa3e P 64aee4950dd845c6b9174d260e230a36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 05:17:21.992961 2786 log.cc:1196] T 881106352e1a42469017cb679ef4fa3e P 64aee4950dd845c6b9174d260e230a36: Deleting WAL directory at /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/minicluster-data/ts-0/wal/wals/881106352e1a42469017cb679ef4fa3e
I20260928 05:17:21.993468 2786 ts_tablet_manager.cc:1939] T 881106352e1a42469017cb679ef4fa3e P 64aee4950dd845c6b9174d260e230a36: Deleting consensus metadata
I20260928 05:17:21.994515 2115 catalog_manager.cc:4841] TS 64aee4950dd845c6b9174d260e230a36 (127.0.97.1:43645): tablet 881106352e1a42469017cb679ef4fa3e (table test-table [id=49e7ee3690724c7e8d9c2f64deb94bd2]) successfully deleted
I20260928 05:17:21.996434 2788 ts_tablet_manager.cc:1918] T 881106352e1a42469017cb679ef4fa3e P 3de53bd0c5ee445fbfbcb66500285846: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 05:17:21.996554 2788 log.cc:1196] T 881106352e1a42469017cb679ef4fa3e P 3de53bd0c5ee445fbfbcb66500285846: Deleting WAL directory at /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/minicluster-data/ts-2/wal/wals/881106352e1a42469017cb679ef4fa3e
I20260928 05:17:21.996909 2788 ts_tablet_manager.cc:1939] T 881106352e1a42469017cb679ef4fa3e P 3de53bd0c5ee445fbfbcb66500285846: Deleting consensus metadata
I20260928 05:17:21.997723 2117 catalog_manager.cc:4841] TS 3de53bd0c5ee445fbfbcb66500285846 (127.0.97.3:43981): tablet 881106352e1a42469017cb679ef4fa3e (table test-table [id=49e7ee3690724c7e8d9c2f64deb94bd2]) successfully deleted
I20260928 05:17:21.998116 2787 ts_tablet_manager.cc:1918] T 881106352e1a42469017cb679ef4fa3e P af01fe7dfaeb464392cfef10732beb0d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 05:17:21.998236 2787 log.cc:1196] T 881106352e1a42469017cb679ef4fa3e P af01fe7dfaeb464392cfef10732beb0d: Deleting WAL directory at /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572622576160-388-0/minicluster-data/ts-1/wal/wals/881106352e1a42469017cb679ef4fa3e
I20260928 05:17:21.998594 2787 ts_tablet_manager.cc:1939] T 881106352e1a42469017cb679ef4fa3e P af01fe7dfaeb464392cfef10732beb0d: Deleting consensus metadata
I20260928 05:17:21.999485 2116 catalog_manager.cc:4841] TS af01fe7dfaeb464392cfef10732beb0d (127.0.97.2:37821): tablet 881106352e1a42469017cb679ef4fa3e (table test-table [id=49e7ee3690724c7e8d9c2f64deb94bd2]) successfully deleted
I20260928 05:17:22.000639 2790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:21.954160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57326 (local address 127.0.97.1:43645)
0928 05:17:21.976588 (+ 22428us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:21.976595 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:21.976620 (+ 25us) server_negotiation.cc:407] Connection header received
0928 05:17:21.976684 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:21.976690 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:21.976780 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:21.977010 (+ 230us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:21.978948 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:21.980151 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:21.982157 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:21.982820 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:21.982936 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:21.994300 (+ 11364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:21.994319 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:21.994340 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:21.994383 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:21.998034 (+ 3651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:21.998714 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:21.998723 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:21.998727 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:21.998881 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:21.999221 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:21.999225 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:21.999228 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:21.999578 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:21.999823 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:22.000242 (+ 419us) server_negotiation.cc:299] Negotiation successful
0928 05:17:22.000396 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22279,"thread_start_us":22708,"threads_started":1}
W20260928 05:17:22.001235 2332 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57326
W20260928 05:17:22.001778 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 2202: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 05:17:22.001951 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 2202
I20260928 05:17:22.021620 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 2341
Sep 28 05:17:22 dist-test-slave-whsn krb5kdc[2086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572640, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260928 05:17:22.034837 2791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:22.022285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46314 (local address 127.0.97.2:37821)
0928 05:17:22.022791 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:22.022795 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:22.022811 (+ 16us) server_negotiation.cc:407] Connection header received
0928 05:17:22.022860 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:22.022864 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:22.022918 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:22.023004 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:22.024079 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:22.024789 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:22.025965 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:22.026350 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:22.026440 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:22.030126 (+ 3686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:22.030139 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:22.030155 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:22.030189 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:22.032723 (+ 2534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:22.033301 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:22.033308 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:22.033310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:22.033438 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:22.033699 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:22.033702 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:22.033704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:22.034019 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:22.034224 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:22.034416 (+ 192us) server_negotiation.cc:299] Negotiation successful
0928 05:17:22.034582 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":270,"threads_started":1}
W20260928 05:17:22.035454 2471 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46314
W20260928 05:17:22.035975 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 2341: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 05:17:22.036120 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 2341
I20260928 05:17:22.049741 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 2480
Sep 28 05:17:22 dist-test-slave-whsn krb5kdc[2086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572640, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260928 05:17:22.063490 2792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:22.050411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37912 (local address 127.0.97.3:43981)
0928 05:17:22.050885 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:22.050890 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:22.050906 (+ 16us) server_negotiation.cc:407] Connection header received
0928 05:17:22.050954 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:22.050958 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:22.051018 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:22.051104 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:22.052223 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:22.052952 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:22.054221 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:22.054629 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:22.054725 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:22.058579 (+ 3854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:22.058600 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:22.058615 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:22.058646 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:22.061149 (+ 2503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:22.061766 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:22.061773 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:22.061776 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:22.061895 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:22.062174 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:22.062178 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:22.062181 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:22.062590 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:22.062811 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:22.063000 (+ 189us) server_negotiation.cc:299] Negotiation successful
0928 05:17:22.063161 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":257,"threads_started":1}
W20260928 05:17:22.064073 2610 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37912
W20260928 05:17:22.064615 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 2480: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 05:17:22.064760 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 2480
I20260928 05:17:22.082062 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 2099
I20260928 05:17:22.083073 2171 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:17:22.194324 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 2099
2026-09-28T05:17:22Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (7888 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572622576160-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:17:22 dist-test-slave-whsn krb5kdc[2796](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:17:22 dist-test-slave-whsn krb5kdc[2796](info): set up 2 sockets
Sep 28 05:17:22 dist-test-slave-whsn krb5kdc[2796](info): commencing operation
krb5kdc: starting...
W20260928 05:17:23.192705 388 mini_kdc.cc:121] Time spent starting KDC: real 0.961s user 0.003s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572622576160-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572622576160-388-0/krb5kdc/test-user.keytab.
Sep 28 05:17:23 dist-test-slave-whsn krb5kdc[2796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572643, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:17:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:17:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 05:17:23.348018 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572622576160-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572622576160-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572622576160-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45549
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45549
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:17:23.639357 2809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:17:23.639598 2809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:17:23.639705 2809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:17:23.646495 2809 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:17:23.646584 2809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:17:23.646634 2809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:17:23.646705 2809 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:17:23.646760 2809 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:17:23.655771 2809 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572622576160-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572622576160-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45549
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572622576160-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:17:23.656922 2809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:17:23.658139 2809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:17:23.669147 2814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:23.669170 2817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:23.669232 2815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:17:23.670135 2809 server_base.cc:1034] running on GCE node
I20260928 05:17:23.670832 2809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:17:23.672145 2809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:17:23.673341 2809 hybrid_clock.cc:648] HybridClock initialized: now 1790572643673298 us; error 30 us; skew 500 ppm
I20260928 05:17:23.673647 2809 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:17:23 dist-test-slave-whsn krb5kdc[2796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572643, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:17:23.676324 2809 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 05:17:23.677671 2809 webserver.cc:460] Webserver started at http://127.0.97.62:42341/ using document root <none> and password file <none>
I20260928 05:17:23.678309 2809 fs_manager.cc:359] Metadata directory not provided
I20260928 05:17:23.678419 2809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:17:23.678697 2809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:17:23.681488 2809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572622576160-388-0/minicluster-data/master-0/data/instance:
uuid: "942de8fdc8be47888632a74f38f4d16d"
format_stamp: "Formatted at 2026-09-28 05:17:23 on dist-test-slave-whsn"
server_key: "cf740a8508866a26b5174bb87e1b184d"
server_key_iv: "a88b3ff215522683fdaebf0d9702cbe0"
server_key_version: "encryptionkey@0"
I20260928 05:17:23.682194 2809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572622576160-388-0/minicluster-data/master-0/wal/instance:
uuid: "942de8fdc8be47888632a74f38f4d16d"
format_stamp: "Formatted at 2026-09-28 05:17:23 on dist-test-slave-whsn"
server_key: "cf740a8508866a26b5174bb87e1b184d"
server_key_iv: "a88b3ff215522683fdaebf0d9702cbe0"
server_key_version: "encryptionkey@0"
I20260928 05:17:23.686764 2809 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 05:17:23.689805 2824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:23.690963 2809 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 05:17:23.691118 2809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572622576160-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572622576160-388-0/minicluster-data/master-0/wal
uuid: "942de8fdc8be47888632a74f38f4d16d"
format_stamp: "Formatted at 2026-09-28 05:17:23 on dist-test-slave-whsn"
server_key: "cf740a8508866a26b5174bb87e1b184d"
server_key_iv: "a88b3ff215522683fdaebf0d9702cbe0"
server_key_version: "encryptionkey@0"
I20260928 05:17:23.691272 2809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572622576160-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572622576160-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572622576160-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:17:23.710132 2809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:17:23.713904 2809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:17:23.714154 2809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:17:23.730284 2809 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45549
I20260928 05:17:23.730317 2875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45549 every 8 connection(s)
I20260928 05:17:23.731494 2809 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:17:23.734856 2876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:17:23.740554 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 2809
I20260928 05:17:23.740764 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572622576160-388-0/minicluster-data/master-0/wal/instance
I20260928 05:17:23.741523 388 external_mini_cluster.cc:1428] Setting key e55e20af22ac400c9f3d619254313267
I20260928 05:17:23.745208 2876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 942de8fdc8be47888632a74f38f4d16d: Bootstrap starting.
I20260928 05:17:23.747947 2876 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 942de8fdc8be47888632a74f38f4d16d: Neither blocks nor log segments found. Creating new log.
I20260928 05:17:23.748942 2876 log.cc:824] T 00000000000000000000000000000000 P 942de8fdc8be47888632a74f38f4d16d: Log is configured to *not* fsync() on all Append() calls
I20260928 05:17:23.751502 2876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 942de8fdc8be47888632a74f38f4d16d: No bootstrap required, opened a new log
Sep 28 05:17:23 dist-test-slave-whsn krb5kdc[2796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572643, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:17:23.759163 2876 raft_consensus.cc:348] T 00000000000000000000000000000000 P 942de8fdc8be47888632a74f38f4d16d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "942de8fdc8be47888632a74f38f4d16d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45549 } }
I20260928 05:17:23.759472 2876 raft_consensus.cc:374] T 00000000000000000000000000000000 P 942de8fdc8be47888632a74f38f4d16d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:17:23.759552 2876 raft_consensus.cc:729] T 00000000000000000000000000000000 P 942de8fdc8be47888632a74f38f4d16d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 942de8fdc8be47888632a74f38f4d16d, State: Initialized, Role: FOLLOWER
I20260928 05:17:23.760043 2876 consensus_queue.cc:260] T 00000000000000000000000000000000 P 942de8fdc8be47888632a74f38f4d16d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "942de8fdc8be47888632a74f38f4d16d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45549 } }
I20260928 05:17:23.760267 2876 raft_consensus.cc:388] T 00000000000000000000000000000000 P 942de8fdc8be47888632a74f38f4d16d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:17:23.760416 2876 raft_consensus.cc:482] T 00000000000000000000000000000000 P 942de8fdc8be47888632a74f38f4d16d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:17:23.760568 2876 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 942de8fdc8be47888632a74f38f4d16d [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:17:23.762099 2879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:23.743517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60636 (local address 127.0.97.62:45549)
0928 05:17:23.744194 (+ 677us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:23.744205 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:23.744242 (+ 37us) server_negotiation.cc:407] Connection header received
0928 05:17:23.745870 (+ 1628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:23.745890 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:23.746241 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:23.746588 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:23.747930 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:23.749275 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:23.750841 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:23.751452 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:23.751613 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:23.755667 (+ 4054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:23.755674 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:23.755689 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:23.755723 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:23.758517 (+ 2794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:23.759097 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:23.759102 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:23.759108 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:23.759260 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:23.759585 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:23.759589 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:23.759592 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:23.760126 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:23.760381 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:23.760790 (+ 409us) server_negotiation.cc:299] Negotiation successful
0928 05:17:23.761178 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":304,"threads_started":1}
I20260928 05:17:23.763062 2876 raft_consensus.cc:504] T 00000000000000000000000000000000 P 942de8fdc8be47888632a74f38f4d16d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "942de8fdc8be47888632a74f38f4d16d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45549 } }
I20260928 05:17:23.763531 2876 leader_election.cc:302] T 00000000000000000000000000000000 P 942de8fdc8be47888632a74f38f4d16d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 942de8fdc8be47888632a74f38f4d16d; no voters:
I20260928 05:17:23.764074 2876 leader_election.cc:288] T 00000000000000000000000000000000 P 942de8fdc8be47888632a74f38f4d16d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:17:23.764176 2881 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 942de8fdc8be47888632a74f38f4d16d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:17:23.765170 2881 raft_consensus.cc:686] T 00000000000000000000000000000000 P 942de8fdc8be47888632a74f38f4d16d [term 1 LEADER]: Becoming Leader. State: Replica: 942de8fdc8be47888632a74f38f4d16d, State: Running, Role: LEADER
I20260928 05:17:23.765693 2881 consensus_queue.cc:237] T 00000000000000000000000000000000 P 942de8fdc8be47888632a74f38f4d16d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "942de8fdc8be47888632a74f38f4d16d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45549 } }
I20260928 05:17:23.766088 2876 sys_catalog.cc:564] T 00000000000000000000000000000000 P 942de8fdc8be47888632a74f38f4d16d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:17:23.768218 2883 sys_catalog.cc:455] T 00000000000000000000000000000000 P 942de8fdc8be47888632a74f38f4d16d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 942de8fdc8be47888632a74f38f4d16d. Latest consensus state: current_term: 1 leader_uuid: "942de8fdc8be47888632a74f38f4d16d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "942de8fdc8be47888632a74f38f4d16d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45549 } } }
I20260928 05:17:23.768539 2883 sys_catalog.cc:458] T 00000000000000000000000000000000 P 942de8fdc8be47888632a74f38f4d16d [sys.catalog]: This master's current role is: LEADER
I20260928 05:17:23.769058 2882 sys_catalog.cc:455] T 00000000000000000000000000000000 P 942de8fdc8be47888632a74f38f4d16d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "942de8fdc8be47888632a74f38f4d16d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "942de8fdc8be47888632a74f38f4d16d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45549 } } }
I20260928 05:17:23.769797 2882 sys_catalog.cc:458] T 00000000000000000000000000000000 P 942de8fdc8be47888632a74f38f4d16d [sys.catalog]: This master's current role is: LEADER
I20260928 05:17:23.771044 2888 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:17:23.775965 2888 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:17:23.784662 2888 catalog_manager.cc:1347] Generated new cluster ID: ef3d6bdc924845f5961a50360c181ffd
I20260928 05:17:23.784763 2888 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:17:23.810133 2888 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:17:23.811864 2888 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:17:23.821341 2888 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 942de8fdc8be47888632a74f38f4d16d: Generated new TSK 0
I20260928 05:17:23.822229 2888 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572622576160-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572622576160-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260928 05:17:23.876179 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572622576160-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572622576160-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572622576160-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572622576160-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45549
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572622576160-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:17:24.177789 2902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:17:24.178040 2902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:17:24.178140 2902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:17:24.184978 2902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:17:24.185068 2902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:17:24.185179 2902 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 05:17:24.194160 2902 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572622576160-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572622576160-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572622576160-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572622576160-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45549
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572622576160-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:17:24.195297 2902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:17:24.196496 2902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:17:24.208425 2907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:24.208448 2908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:24.208487 2910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:17:24.209164 2902 server_base.cc:1034] running on GCE node
I20260928 05:17:24.209661 2902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:17:24.210541 2902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:17:24.211722 2902 hybrid_clock.cc:648] HybridClock initialized: now 1790572644211681 us; error 34 us; skew 500 ppm
I20260928 05:17:24.211992 2902 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:17:24 dist-test-slave-whsn krb5kdc[2796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572644, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:17:24.214746 2902 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260928 05:17:24.215950 2902 webserver.cc:460] Webserver started at http://127.0.97.1:39745/ using document root <none> and password file <none>
I20260928 05:17:24.216529 2902 fs_manager.cc:359] Metadata directory not provided
I20260928 05:17:24.216627 2902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:17:24.216928 2902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:17:24.219621 2902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572622576160-388-0/minicluster-data/ts-0/data/instance:
uuid: "2365c83045f744318665cf458cd569dc"
format_stamp: "Formatted at 2026-09-28 05:17:24 on dist-test-slave-whsn"
server_key: "4aca6b55a6213ef90139aeb10c035cb5"
server_key_iv: "6e19834c02e936d513fa7b84b2d2765b"
server_key_version: "encryptionkey@0"
I20260928 05:17:24.220279 2902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572622576160-388-0/minicluster-data/ts-0/wal/instance:
uuid: "2365c83045f744318665cf458cd569dc"
format_stamp: "Formatted at 2026-09-28 05:17:24 on dist-test-slave-whsn"
server_key: "4aca6b55a6213ef90139aeb10c035cb5"
server_key_iv: "6e19834c02e936d513fa7b84b2d2765b"
server_key_version: "encryptionkey@0"
I20260928 05:17:24.224767 2902 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260928 05:17:24.227816 2917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:24.229041 2902 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 05:17:24.229158 2902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572622576160-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572622576160-388-0/minicluster-data/ts-0/wal
uuid: "2365c83045f744318665cf458cd569dc"
format_stamp: "Formatted at 2026-09-28 05:17:24 on dist-test-slave-whsn"
server_key: "4aca6b55a6213ef90139aeb10c035cb5"
server_key_iv: "6e19834c02e936d513fa7b84b2d2765b"
server_key_version: "encryptionkey@0"
I20260928 05:17:24.229307 2902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572622576160-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572622576160-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572622576160-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:17:24.260998 2902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:17:24.264895 2902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:17:24.265121 2902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:17:24.265975 2902 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:17:24.267217 2902 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:17:24.267285 2902 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:24.267393 2902 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:17:24.267437 2902 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:24.295128 2902 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38055
I20260928 05:17:24.295172 3029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38055 every 8 connection(s)
I20260928 05:17:24.296288 2902 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572622576160-388-0/minicluster-data/ts-0/data/info.pb
I20260928 05:17:24.301811 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 2902
I20260928 05:17:24.302007 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572622576160-388-0/minicluster-data/ts-0/wal/instance
I20260928 05:17:24.302597 388 external_mini_cluster.cc:1428] Setting key 60e0417f8c0b14d32b13849b2629769f
Sep 28 05:17:24 dist-test-slave-whsn krb5kdc[2796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572644, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:17:24.314847 3032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:24.298569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51861 (local address 127.0.97.62:45549)
0928 05:17:24.298933 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:24.298937 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:24.299856 (+ 919us) server_negotiation.cc:407] Connection header received
0928 05:17:24.301626 (+ 1770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:24.301632 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:24.301717 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:24.301842 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:24.303244 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:24.303973 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:24.305498 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:24.305880 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:24.305985 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:24.309133 (+ 3148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:24.309139 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:24.309143 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:24.309176 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:24.312048 (+ 2872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:24.312684 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:24.312688 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:24.312692 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:24.312816 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:24.313240 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:24.313244 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:24.313247 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:24.313563 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:24.313698 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:24.314275 (+ 577us) server_negotiation.cc:299] Negotiation successful
0928 05:17:24.314478 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":177,"threads_started":1}
I20260928 05:17:24.315593 3033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:24.299096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45549 (local address 127.0.97.1:51861)
0928 05:17:24.299701 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:24.299740 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:24.301358 (+ 1618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:24.302058 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:24.302070 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:24.302537 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:17:24.303088 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:24.303105 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:24.304113 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:24.304117 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:24.305367 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:24.305378 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:24.306088 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:24.306096 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:24.306295 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:24.306859 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:17:24.306884 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:17:24.308942 (+ 2058us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:17:24.312176 (+ 3234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:24.312182 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:24.312193 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:24.312575 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:24.312949 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:24.312952 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:24.312955 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:24.313130 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:24.313670 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:17:24.313676 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:17:24.313954 (+ 278us) client_negotiation.cc:769] Sending connection context
0928 05:17:24.314164 (+ 210us) client_negotiation.cc:240] Negotiation successful
0928 05:17:24.314553 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":244,"threads_started":1}
I20260928 05:17:24.316991 3030 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45549
I20260928 05:17:24.317301 3030 heartbeater.cc:464] Registering TS with master...
I20260928 05:17:24.318137 3030 heartbeater.cc:511] Master 127.0.97.62:45549 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260928 05:17:24.320312 2841 ts_manager.cc:194] Registered new tserver with Master: 2365c83045f744318665cf458cd569dc (127.0.97.1:38055)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260928 05:17:24.322203 2841 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:51861
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572622576160-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572622576160-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260928 05:17:24.339207 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572622576160-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572622576160-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572622576160-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572622576160-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45549
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572622576160-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:17:24.635205 3036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:17:24.635470 3036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:17:24.635592 3036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:17:24.642474 3036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:17:24.642583 3036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:17:24.642732 3036 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 05:17:24.651903 3036 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572622576160-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572622576160-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572622576160-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572622576160-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45549
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572622576160-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:17:24.653111 3036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:17:24.654307 3036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:17:24.666188 3041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:24.666275 3042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:24.666326 3044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:17:24.666898 3036 server_base.cc:1034] running on GCE node
I20260928 05:17:24.667400 3036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:17:24.668229 3036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:17:24.669402 3036 hybrid_clock.cc:648] HybridClock initialized: now 1790572644669351 us; error 44 us; skew 500 ppm
I20260928 05:17:24.669661 3036 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:17:24 dist-test-slave-whsn krb5kdc[2796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572644, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:17:24.672456 3036 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260928 05:17:24.673858 3036 webserver.cc:460] Webserver started at http://127.0.97.2:45687/ using document root <none> and password file <none>
I20260928 05:17:24.674520 3036 fs_manager.cc:359] Metadata directory not provided
I20260928 05:17:24.674640 3036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:17:24.674906 3036 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:17:24.677548 3036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572622576160-388-0/minicluster-data/ts-1/data/instance:
uuid: "4b6ed4530e5a49339a142632b947b2d9"
format_stamp: "Formatted at 2026-09-28 05:17:24 on dist-test-slave-whsn"
server_key: "5ef85a8485bc30e4e387f9846f6030d6"
server_key_iv: "f059cdbc357652f230df624589bb86f3"
server_key_version: "encryptionkey@0"
I20260928 05:17:24.678243 3036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572622576160-388-0/minicluster-data/ts-1/wal/instance:
uuid: "4b6ed4530e5a49339a142632b947b2d9"
format_stamp: "Formatted at 2026-09-28 05:17:24 on dist-test-slave-whsn"
server_key: "5ef85a8485bc30e4e387f9846f6030d6"
server_key_iv: "f059cdbc357652f230df624589bb86f3"
server_key_version: "encryptionkey@0"
I20260928 05:17:24.682775 3036 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 05:17:24.686192 3051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:24.687445 3036 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:17:24.687579 3036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572622576160-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572622576160-388-0/minicluster-data/ts-1/wal
uuid: "4b6ed4530e5a49339a142632b947b2d9"
format_stamp: "Formatted at 2026-09-28 05:17:24 on dist-test-slave-whsn"
server_key: "5ef85a8485bc30e4e387f9846f6030d6"
server_key_iv: "f059cdbc357652f230df624589bb86f3"
server_key_version: "encryptionkey@0"
I20260928 05:17:24.687754 3036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572622576160-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572622576160-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572622576160-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:17:24.709398 3036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:17:24.712901 3036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:17:24.713142 3036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:17:24.713972 3036 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:17:24.715178 3036 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:17:24.715248 3036 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:24.715349 3036 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:17:24.715394 3036 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:24.744482 3036 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34441
I20260928 05:17:24.744526 3163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34441 every 8 connection(s)
I20260928 05:17:24.745674 3036 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572622576160-388-0/minicluster-data/ts-1/data/info.pb
I20260928 05:17:24.752206 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 3036
I20260928 05:17:24.752394 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572622576160-388-0/minicluster-data/ts-1/wal/instance
I20260928 05:17:24.752810 388 external_mini_cluster.cc:1428] Setting key 74d270aeaf961acec9add3ae454a1afc
Sep 28 05:17:24 dist-test-slave-whsn krb5kdc[2796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572644, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:17:24.764237 3032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:24.747966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39095 (local address 127.0.97.62:45549)
0928 05:17:24.748128 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:24.748132 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:24.749160 (+ 1028us) server_negotiation.cc:407] Connection header received
0928 05:17:24.750847 (+ 1687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:24.750852 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:24.750930 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:24.751023 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:24.752390 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:24.753222 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:24.754669 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:24.755050 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:24.755153 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:24.758658 (+ 3505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:24.758663 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:24.758666 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:24.758694 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:24.761597 (+ 2903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:24.762263 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:24.762268 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:24.762272 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:24.762410 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:24.762758 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:24.762762 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:24.762764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:24.763058 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:24.763187 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:24.763735 (+ 548us) server_negotiation.cc:299] Negotiation successful
0928 05:17:24.763923 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 05:17:24.765026 3166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:24.748338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45549 (local address 127.0.97.2:39095)
0928 05:17:24.749014 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:24.749053 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:24.750607 (+ 1554us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:24.751234 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:24.751242 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:24.751701 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:17:24.752233 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:24.752253 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:24.753356 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:24.753360 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:24.754536 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:24.754548 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:24.755244 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:24.755251 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:24.755452 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:24.756092 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:17:24.756112 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:17:24.758452 (+ 2340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:17:24.761728 (+ 3276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:24.761734 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:24.761745 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:24.762165 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:24.762501 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:24.762504 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:24.762507 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:24.762674 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:24.763158 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:17:24.763163 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:17:24.763426 (+ 263us) client_negotiation.cc:769] Sending connection context
0928 05:17:24.763637 (+ 211us) client_negotiation.cc:240] Negotiation successful
0928 05:17:24.763972 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":235,"threads_started":1}
I20260928 05:17:24.766304 3164 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45549
I20260928 05:17:24.766568 3164 heartbeater.cc:464] Registering TS with master...
I20260928 05:17:24.767192 3164 heartbeater.cc:511] Master 127.0.97.62:45549 requested a full tablet report, sending...
I20260928 05:17:24.768499 2841 ts_manager.cc:194] Registered new tserver with Master: 4b6ed4530e5a49339a142632b947b2d9 (127.0.97.2:34441)
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260928 05:17:24.769317 2841 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:39095
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572622576160-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572622576160-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260928 05:17:24.789691 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572622576160-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572622576160-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572622576160-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572622576160-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45549
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572622576160-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:17:25.090368 3169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:17:25.090675 3169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:17:25.090816 3169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:17:25.098047 3169 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:17:25.098157 3169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:17:25.098304 3169 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 05:17:25.107757 3169 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572622576160-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572622576160-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572622576160-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572622576160-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45549
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572622576160-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:17:25.109056 3169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:17:25.110309 3169 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:17:25.122488 3175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:25.122714 3177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:25.123133 3174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:17:25.123172 3169 server_base.cc:1034] running on GCE node
I20260928 05:17:25.123775 3169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:17:25.124667 3169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:17:25.125854 3169 hybrid_clock.cc:648] HybridClock initialized: now 1790572645125785 us; error 65 us; skew 500 ppm
I20260928 05:17:25.126142 3169 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:17:25 dist-test-slave-whsn krb5kdc[2796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572645, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:17:25.128984 3169 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260928 05:17:25.130288 3169 webserver.cc:460] Webserver started at http://127.0.97.3:40299/ using document root <none> and password file <none>
I20260928 05:17:25.130911 3169 fs_manager.cc:359] Metadata directory not provided
I20260928 05:17:25.131112 3169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:17:25.131444 3169 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:17:25.134478 3169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572622576160-388-0/minicluster-data/ts-2/data/instance:
uuid: "f7ea7afa10b94041839bfdccecab2066"
format_stamp: "Formatted at 2026-09-28 05:17:25 on dist-test-slave-whsn"
server_key: "2f2807622dd39d56d424a1c231b68180"
server_key_iv: "07b4a3476991c4bc04af35cda95c10d5"
server_key_version: "encryptionkey@0"
I20260928 05:17:25.135461 3169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572622576160-388-0/minicluster-data/ts-2/wal/instance:
uuid: "f7ea7afa10b94041839bfdccecab2066"
format_stamp: "Formatted at 2026-09-28 05:17:25 on dist-test-slave-whsn"
server_key: "2f2807622dd39d56d424a1c231b68180"
server_key_iv: "07b4a3476991c4bc04af35cda95c10d5"
server_key_version: "encryptionkey@0"
I20260928 05:17:25.140828 3169 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 05:17:25.144218 3184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:25.145539 3169 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:17:25.145697 3169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572622576160-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572622576160-388-0/minicluster-data/ts-2/wal
uuid: "f7ea7afa10b94041839bfdccecab2066"
format_stamp: "Formatted at 2026-09-28 05:17:25 on dist-test-slave-whsn"
server_key: "2f2807622dd39d56d424a1c231b68180"
server_key_iv: "07b4a3476991c4bc04af35cda95c10d5"
server_key_version: "encryptionkey@0"
I20260928 05:17:25.145895 3169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572622576160-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572622576160-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572622576160-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:17:25.159900 3169 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:17:25.163729 3169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:17:25.164005 3169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:17:25.164935 3169 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:17:25.166256 3169 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:17:25.166340 3169 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:25.166477 3169 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:17:25.166549 3169 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:25.197846 3169 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43551
I20260928 05:17:25.197882 3296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43551 every 8 connection(s)
I20260928 05:17:25.199127 3169 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572622576160-388-0/minicluster-data/ts-2/data/info.pb
I20260928 05:17:25.204427 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 3169
I20260928 05:17:25.204581 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572622576160-388-0/minicluster-data/ts-2/wal/instance
I20260928 05:17:25.205153 388 external_mini_cluster.cc:1428] Setting key 05022d4807f9b77cfe0e8be81b9cabaa
Sep 28 05:17:25 dist-test-slave-whsn krb5kdc[2796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572645, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:17:25.217693 3032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:25.201527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34235 (local address 127.0.97.62:45549)
0928 05:17:25.201724 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:25.201729 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:25.202812 (+ 1083us) server_negotiation.cc:407] Connection header received
0928 05:17:25.204519 (+ 1707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:25.204525 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:25.204608 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:25.204730 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:25.206255 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:25.207155 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:25.208711 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:25.209172 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:25.209263 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:25.212203 (+ 2940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:25.212208 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:25.212211 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:25.212242 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:25.214924 (+ 2682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:25.215566 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:25.215572 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:25.215576 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:25.215690 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:25.216073 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:25.216078 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:25.216082 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:25.216338 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:25.216462 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:25.217160 (+ 698us) server_negotiation.cc:299] Negotiation successful
0928 05:17:25.217362 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 05:17:25.218741 3299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:25.201957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45549 (local address 127.0.97.3:34235)
0928 05:17:25.202652 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:25.202693 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:25.204258 (+ 1565us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:25.205021 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:25.205034 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:25.205571 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:17:25.206113 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:25.206130 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:25.207290 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:25.207294 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:25.208578 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:25.208591 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:25.209332 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:25.209336 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:25.209528 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:25.210074 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:17:25.210094 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:17:25.212016 (+ 1922us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:17:25.215067 (+ 3051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:25.215075 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:25.215091 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:25.215457 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:25.215786 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:25.215791 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:25.215795 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:25.215967 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:25.216452 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:17:25.216459 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:17:25.216776 (+ 317us) client_negotiation.cc:769] Sending connection context
0928 05:17:25.217045 (+ 269us) client_negotiation.cc:240] Negotiation successful
0928 05:17:25.217527 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":263,"threads_started":1}
I20260928 05:17:25.220002 3297 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45549
I20260928 05:17:25.220269 3297 heartbeater.cc:464] Registering TS with master...
I20260928 05:17:25.220958 3297 heartbeater.cc:511] Master 127.0.97.62:45549 requested a full tablet report, sending...
I20260928 05:17:25.222210 2840 ts_manager.cc:194] Registered new tserver with Master: f7ea7afa10b94041839bfdccecab2066 (127.0.97.3:43551)
I20260928 05:17:25.223032 2840 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:34235
I20260928 05:17:25.231902 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 05:17:25.258608 3032 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:17:25.256420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60650 (local address 127.0.97.62:45549)
0928 05:17:25.256591 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:25.256595 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:25.257050 (+ 455us) server_negotiation.cc:407] Connection header received
0928 05:17:25.257190 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:25.257194 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:25.257255 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:25.257346 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:25.258412 (+ 1066us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60650: BlockingRecv error: recv got EOF from 127.0.0.1:60650 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 05:17:25.262017 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 2902
W20260928 05:17:25.264272 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 2902: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:38055: server requires authentication, but client does not have Kerberos credentials available
I20260928 05:17:25.264444 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 2902
W20260928 05:17:25.264473 3308 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:17:25.262771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54380 (local address 127.0.97.1:38055)
0928 05:17:25.263222 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:25.263230 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:25.263254 (+ 24us) server_negotiation.cc:407] Connection header received
0928 05:17:25.263305 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:25.263311 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:25.263525 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:25.263650 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:25.264172 (+ 522us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54380: BlockingRecv error: recv got EOF from 127.0.0.1:54380 (error 108)
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":221,"threads_started":1}
I20260928 05:17:25.277330 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 3036
W20260928 05:17:25.279632 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 3036: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:34441: server requires authentication, but client does not have Kerberos credentials available
I20260928 05:17:25.279762 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 3036
I20260928 05:17:25.291517 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 3169
W20260928 05:17:25.294252 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 3169: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:43551: server requires authentication, but client does not have Kerberos credentials available
I20260928 05:17:25.294389 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 3169
I20260928 05:17:25.305969 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 2809
I20260928 05:17:25.307005 2871 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:17:25.410113 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 2809
2026-09-28T05:17:25Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3213 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790572622576160-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:17:25 dist-test-slave-whsn krb5kdc[3314](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:17:25 dist-test-slave-whsn krb5kdc[3314](info): set up 2 sockets
Sep 28 05:17:25 dist-test-slave-whsn krb5kdc[3314](info): commencing operation
krb5kdc: starting...
W20260928 05:17:26.400673 388 mini_kdc.cc:121] Time spent starting KDC: real 0.955s user 0.005s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790572622576160-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790572622576160-388-0/krb5kdc/test-user.keytab.
Sep 28 05:17:26 dist-test-slave-whsn krb5kdc[3314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572646, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:17:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:17:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 05:17:26.562481 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790572622576160-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790572622576160-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790572622576160-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42773
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42773
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:17:26.862465 3327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:17:26.862740 3327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:17:26.862877 3327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:17:26.869925 3327 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:17:26.870028 3327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:17:26.870113 3327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:17:26.870194 3327 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:17:26.870265 3327 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:17:26.880048 3327 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790572622576160-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790572622576160-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42773
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42773
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790572622576160-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:17:26.881222 3327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:17:26.882455 3327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:17:26.893376 3332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:26.893359 3333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:26.893474 3335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:17:26.894021 3327 server_base.cc:1034] running on GCE node
I20260928 05:17:26.894699 3327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:17:26.895895 3327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:17:26.897073 3327 hybrid_clock.cc:648] HybridClock initialized: now 1790572646897033 us; error 39 us; skew 500 ppm
I20260928 05:17:26.897370 3327 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:17:26 dist-test-slave-whsn krb5kdc[3314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572646, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:17:26.900106 3327 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 05:17:26.901373 3327 webserver.cc:460] Webserver started at http://127.0.97.62:38083/ using document root <none> and password file <none>
I20260928 05:17:26.901954 3327 fs_manager.cc:359] Metadata directory not provided
I20260928 05:17:26.902064 3327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:17:26.902311 3327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:17:26.905009 3327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790572622576160-388-0/minicluster-data/master-0/data/instance:
uuid: "f25c5ae33b5d407cbf1dd5c43b19df0b"
format_stamp: "Formatted at 2026-09-28 05:17:26 on dist-test-slave-whsn"
server_key: "5074478d7a0f25de3480e91c76ef6361"
server_key_iv: "a6d04d0daf4a76effecf1e6c1c1e8bcf"
server_key_version: "encryptionkey@0"
I20260928 05:17:26.905706 3327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790572622576160-388-0/minicluster-data/master-0/wal/instance:
uuid: "f25c5ae33b5d407cbf1dd5c43b19df0b"
format_stamp: "Formatted at 2026-09-28 05:17:26 on dist-test-slave-whsn"
server_key: "5074478d7a0f25de3480e91c76ef6361"
server_key_iv: "a6d04d0daf4a76effecf1e6c1c1e8bcf"
server_key_version: "encryptionkey@0"
I20260928 05:17:26.910163 3327 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260928 05:17:26.913242 3342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:26.914407 3327 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.002s
I20260928 05:17:26.914557 3327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790572622576160-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790572622576160-388-0/minicluster-data/master-0/wal
uuid: "f25c5ae33b5d407cbf1dd5c43b19df0b"
format_stamp: "Formatted at 2026-09-28 05:17:26 on dist-test-slave-whsn"
server_key: "5074478d7a0f25de3480e91c76ef6361"
server_key_iv: "a6d04d0daf4a76effecf1e6c1c1e8bcf"
server_key_version: "encryptionkey@0"
I20260928 05:17:26.914705 3327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790572622576160-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790572622576160-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790572622576160-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:17:26.936359 3327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:17:26.940119 3327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:17:26.940351 3327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:17:26.956144 3327 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42773
I20260928 05:17:26.956179 3393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42773 every 8 connection(s)
I20260928 05:17:26.957304 3327 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:17:26.960536 3394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:17:26.963833 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 3327
I20260928 05:17:26.964022 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790572622576160-388-0/minicluster-data/master-0/wal/instance
I20260928 05:17:26.964576 388 external_mini_cluster.cc:1428] Setting key 7a5e6da750250ff41eaac3365cc5494b
I20260928 05:17:26.970487 3394 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f25c5ae33b5d407cbf1dd5c43b19df0b: Bootstrap starting.
I20260928 05:17:26.973750 3394 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f25c5ae33b5d407cbf1dd5c43b19df0b: Neither blocks nor log segments found. Creating new log.
I20260928 05:17:26.974788 3394 log.cc:824] T 00000000000000000000000000000000 P f25c5ae33b5d407cbf1dd5c43b19df0b: Log is configured to *not* fsync() on all Append() calls
I20260928 05:17:26.977643 3394 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f25c5ae33b5d407cbf1dd5c43b19df0b: No bootstrap required, opened a new log
Sep 28 05:17:26 dist-test-slave-whsn krb5kdc[3314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572646, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:17:26.985764 3397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:26.966209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49652 (local address 127.0.97.62:42773)
0928 05:17:26.966947 (+ 738us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:26.966958 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:26.967002 (+ 44us) server_negotiation.cc:407] Connection header received
0928 05:17:26.969095 (+ 2093us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:26.969117 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:26.969459 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:26.969756 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:26.971406 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:26.972667 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:26.974331 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:26.974975 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:26.975105 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:26.979363 (+ 4258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:26.979369 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:26.979382 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:26.979415 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:26.982212 (+ 2797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:26.982936 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:26.982942 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:26.982947 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:26.983101 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:26.983431 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:26.983434 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:26.983436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:26.983898 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:26.984127 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:26.984477 (+ 350us) server_negotiation.cc:299] Negotiation successful
0928 05:17:26.984806 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":339,"threads_started":1}
I20260928 05:17:26.985992 3394 raft_consensus.cc:348] T 00000000000000000000000000000000 P f25c5ae33b5d407cbf1dd5c43b19df0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f25c5ae33b5d407cbf1dd5c43b19df0b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42773 } }
I20260928 05:17:26.986306 3394 raft_consensus.cc:374] T 00000000000000000000000000000000 P f25c5ae33b5d407cbf1dd5c43b19df0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:17:26.986402 3394 raft_consensus.cc:729] T 00000000000000000000000000000000 P f25c5ae33b5d407cbf1dd5c43b19df0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f25c5ae33b5d407cbf1dd5c43b19df0b, State: Initialized, Role: FOLLOWER
I20260928 05:17:26.986994 3394 consensus_queue.cc:260] T 00000000000000000000000000000000 P f25c5ae33b5d407cbf1dd5c43b19df0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f25c5ae33b5d407cbf1dd5c43b19df0b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42773 } }
I20260928 05:17:26.987195 3394 raft_consensus.cc:388] T 00000000000000000000000000000000 P f25c5ae33b5d407cbf1dd5c43b19df0b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:17:26.987316 3394 raft_consensus.cc:482] T 00000000000000000000000000000000 P f25c5ae33b5d407cbf1dd5c43b19df0b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:17:26.987460 3394 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f25c5ae33b5d407cbf1dd5c43b19df0b [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:17:26.989229 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 05:17:26.989936 3394 raft_consensus.cc:504] T 00000000000000000000000000000000 P f25c5ae33b5d407cbf1dd5c43b19df0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f25c5ae33b5d407cbf1dd5c43b19df0b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42773 } }
I20260928 05:17:26.990429 3394 leader_election.cc:302] T 00000000000000000000000000000000 P f25c5ae33b5d407cbf1dd5c43b19df0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f25c5ae33b5d407cbf1dd5c43b19df0b; no voters:
I20260928 05:17:26.991003 3394 leader_election.cc:288] T 00000000000000000000000000000000 P f25c5ae33b5d407cbf1dd5c43b19df0b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:17:26.991084 3399 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f25c5ae33b5d407cbf1dd5c43b19df0b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:17:26.992206 3399 raft_consensus.cc:686] T 00000000000000000000000000000000 P f25c5ae33b5d407cbf1dd5c43b19df0b [term 1 LEADER]: Becoming Leader. State: Replica: f25c5ae33b5d407cbf1dd5c43b19df0b, State: Running, Role: LEADER
I20260928 05:17:26.992699 3399 consensus_queue.cc:237] T 00000000000000000000000000000000 P f25c5ae33b5d407cbf1dd5c43b19df0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f25c5ae33b5d407cbf1dd5c43b19df0b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42773 } }
I20260928 05:17:26.992976 3394 sys_catalog.cc:564] T 00000000000000000000000000000000 P f25c5ae33b5d407cbf1dd5c43b19df0b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:17:26.995095 3401 sys_catalog.cc:455] T 00000000000000000000000000000000 P f25c5ae33b5d407cbf1dd5c43b19df0b [sys.catalog]: SysCatalogTable state changed. Reason: New leader f25c5ae33b5d407cbf1dd5c43b19df0b. Latest consensus state: current_term: 1 leader_uuid: "f25c5ae33b5d407cbf1dd5c43b19df0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f25c5ae33b5d407cbf1dd5c43b19df0b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42773 } } }
I20260928 05:17:26.995399 3401 sys_catalog.cc:458] T 00000000000000000000000000000000 P f25c5ae33b5d407cbf1dd5c43b19df0b [sys.catalog]: This master's current role is: LEADER
I20260928 05:17:26.995918 3400 sys_catalog.cc:455] T 00000000000000000000000000000000 P f25c5ae33b5d407cbf1dd5c43b19df0b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f25c5ae33b5d407cbf1dd5c43b19df0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f25c5ae33b5d407cbf1dd5c43b19df0b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42773 } } }
I20260928 05:17:26.996215 3400 sys_catalog.cc:458] T 00000000000000000000000000000000 P f25c5ae33b5d407cbf1dd5c43b19df0b [sys.catalog]: This master's current role is: LEADER
I20260928 05:17:26.998364 3406 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:17:27.002907 3406 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:17:27.027101 3406 catalog_manager.cc:1347] Generated new cluster ID: 3061081e78f7480db305ae8910b4643a
I20260928 05:17:27.027225 3406 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:17:27.056483 3397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:27.045905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49662 (local address 127.0.97.62:42773)
0928 05:17:27.046094 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:27.046101 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:27.046590 (+ 489us) server_negotiation.cc:407] Connection header received
0928 05:17:27.046759 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:27.046763 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:27.046827 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:27.046910 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:27.047855 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:27.048559 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:27.049834 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:27.050254 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:27.050356 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:27.054997 (+ 4641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:27.055002 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:27.055006 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:17:27.055640 (+ 634us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:27.055704 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:17:27.055740 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:27.055846 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:27.056043 (+ 197us) server_negotiation.cc:299] Negotiation successful
0928 05:17:27.056130 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 05:17:27.057829 3406 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:17:27.059558 3406 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:17:27.070008 3406 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f25c5ae33b5d407cbf1dd5c43b19df0b: Generated new TSK 0
I20260928 05:17:27.070894 3406 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 05:17:27.081208 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 3327
I20260928 05:17:27.082291 3389 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:17:27.170143 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 3327
2026-09-28T05:17:27Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1755 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572622576160-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:17:27 dist-test-slave-whsn krb5kdc[3428](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:17:27 dist-test-slave-whsn krb5kdc[3428](info): set up 2 sockets
Sep 28 05:17:27 dist-test-slave-whsn krb5kdc[3428](info): commencing operation
krb5kdc: starting...
W20260928 05:17:28.160920 388 mini_kdc.cc:121] Time spent starting KDC: real 0.960s user 0.007s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572622576160-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572622576160-388-0/krb5kdc/test-user.keytab.
Sep 28 05:17:28 dist-test-slave-whsn krb5kdc[3428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572648, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:17:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:17:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 05:17:28.318053 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572622576160-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572622576160-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572622576160-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39705
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39705
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:17:28.611840 3441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:17:28.612097 3441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:17:28.612221 3441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:17:28.619020 3441 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:17:28.619113 3441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:17:28.619184 3441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:17:28.619246 3441 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:17:28.619303 3441 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:17:28.628592 3441 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572622576160-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572622576160-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39705
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39705
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572622576160-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:17:28.629750 3441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:17:28.630923 3441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:17:28.641111 3447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:28.641074 3449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:28.641553 3446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:17:28.642001 3441 server_base.cc:1034] running on GCE node
I20260928 05:17:28.642655 3441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:17:28.643842 3441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:17:28.645025 3441 hybrid_clock.cc:648] HybridClock initialized: now 1790572648644985 us; error 25 us; skew 500 ppm
I20260928 05:17:28.645299 3441 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:17:28 dist-test-slave-whsn krb5kdc[3428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572648, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:17:28.647894 3441 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 05:17:28.649228 3441 webserver.cc:460] Webserver started at http://127.0.97.62:32825/ using document root <none> and password file <none>
I20260928 05:17:28.649807 3441 fs_manager.cc:359] Metadata directory not provided
I20260928 05:17:28.649900 3441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:17:28.650137 3441 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:17:28.652750 3441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572622576160-388-0/minicluster-data/master-0/data/instance:
uuid: "57f770d65e0240418c10f071ae3bc1cf"
format_stamp: "Formatted at 2026-09-28 05:17:28 on dist-test-slave-whsn"
server_key: "1270540af73671ba3aa7ca47075013c3"
server_key_iv: "1e8db5474222e88643848d03575c82ba"
server_key_version: "encryptionkey@0"
I20260928 05:17:28.653445 3441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572622576160-388-0/minicluster-data/master-0/wal/instance:
uuid: "57f770d65e0240418c10f071ae3bc1cf"
format_stamp: "Formatted at 2026-09-28 05:17:28 on dist-test-slave-whsn"
server_key: "1270540af73671ba3aa7ca47075013c3"
server_key_iv: "1e8db5474222e88643848d03575c82ba"
server_key_version: "encryptionkey@0"
I20260928 05:17:28.657917 3441 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 05:17:28.660840 3456 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:28.662022 3441 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 05:17:28.662160 3441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572622576160-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572622576160-388-0/minicluster-data/master-0/wal
uuid: "57f770d65e0240418c10f071ae3bc1cf"
format_stamp: "Formatted at 2026-09-28 05:17:28 on dist-test-slave-whsn"
server_key: "1270540af73671ba3aa7ca47075013c3"
server_key_iv: "1e8db5474222e88643848d03575c82ba"
server_key_version: "encryptionkey@0"
I20260928 05:17:28.662312 3441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572622576160-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572622576160-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572622576160-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:17:28.677125 3441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:17:28.680392 3441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:17:28.680624 3441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:17:28.696213 3441 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39705
I20260928 05:17:28.696234 3507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39705 every 8 connection(s)
I20260928 05:17:28.697389 3441 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:17:28.699771 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 3441
I20260928 05:17:28.699963 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572622576160-388-0/minicluster-data/master-0/wal/instance
I20260928 05:17:28.700572 388 external_mini_cluster.cc:1428] Setting key 385a7e20dd1c5b90108de06d2d7a39e9
I20260928 05:17:28.700701 3508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:17:28.713474 3508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57f770d65e0240418c10f071ae3bc1cf: Bootstrap starting.
Sep 28 05:17:28 dist-test-slave-whsn krb5kdc[3428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572648, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:17:28.716688 3508 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 57f770d65e0240418c10f071ae3bc1cf: Neither blocks nor log segments found. Creating new log.
I20260928 05:17:28.717871 3508 log.cc:824] T 00000000000000000000000000000000 P 57f770d65e0240418c10f071ae3bc1cf: Log is configured to *not* fsync() on all Append() calls
I20260928 05:17:28.720783 3508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57f770d65e0240418c10f071ae3bc1cf: No bootstrap required, opened a new log
I20260928 05:17:28.722061 3511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:28.702294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47452 (local address 127.0.97.62:39705)
0928 05:17:28.703031 (+ 737us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:28.703045 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:28.703090 (+ 45us) server_negotiation.cc:407] Connection header received
0928 05:17:28.705059 (+ 1969us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:28.705085 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:28.705540 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:28.705947 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:28.707012 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:28.708909 (+ 1897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:28.710021 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:28.711015 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:28.711182 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:28.714717 (+ 3535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:28.714727 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:28.714744 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:28.714784 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:28.718206 (+ 3422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:28.718930 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:28.718935 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:28.718940 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:28.719089 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:28.719417 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:28.719421 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:28.719424 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:28.720016 (+ 592us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:28.720292 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:28.720518 (+ 226us) server_negotiation.cc:299] Negotiation successful
0928 05:17:28.720908 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":308,"threads_started":1}
I20260928 05:17:28.728595 3508 raft_consensus.cc:348] T 00000000000000000000000000000000 P 57f770d65e0240418c10f071ae3bc1cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57f770d65e0240418c10f071ae3bc1cf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39705 } }
I20260928 05:17:28.728901 3508 raft_consensus.cc:374] T 00000000000000000000000000000000 P 57f770d65e0240418c10f071ae3bc1cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:17:28.728981 3508 raft_consensus.cc:729] T 00000000000000000000000000000000 P 57f770d65e0240418c10f071ae3bc1cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57f770d65e0240418c10f071ae3bc1cf, State: Initialized, Role: FOLLOWER
I20260928 05:17:28.729449 3508 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57f770d65e0240418c10f071ae3bc1cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57f770d65e0240418c10f071ae3bc1cf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39705 } }
I20260928 05:17:28.729631 3508 raft_consensus.cc:388] T 00000000000000000000000000000000 P 57f770d65e0240418c10f071ae3bc1cf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:17:28.729718 3508 raft_consensus.cc:482] T 00000000000000000000000000000000 P 57f770d65e0240418c10f071ae3bc1cf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:17:28.729828 3508 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 57f770d65e0240418c10f071ae3bc1cf [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:17:28.731900 3508 raft_consensus.cc:504] T 00000000000000000000000000000000 P 57f770d65e0240418c10f071ae3bc1cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57f770d65e0240418c10f071ae3bc1cf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39705 } }
I20260928 05:17:28.732307 3508 leader_election.cc:302] T 00000000000000000000000000000000 P 57f770d65e0240418c10f071ae3bc1cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 57f770d65e0240418c10f071ae3bc1cf; no voters:
I20260928 05:17:28.732782 3508 leader_election.cc:288] T 00000000000000000000000000000000 P 57f770d65e0240418c10f071ae3bc1cf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:17:28.732914 3513 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 57f770d65e0240418c10f071ae3bc1cf [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:17:28.733510 3513 raft_consensus.cc:686] T 00000000000000000000000000000000 P 57f770d65e0240418c10f071ae3bc1cf [term 1 LEADER]: Becoming Leader. State: Replica: 57f770d65e0240418c10f071ae3bc1cf, State: Running, Role: LEADER
I20260928 05:17:28.734014 3513 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57f770d65e0240418c10f071ae3bc1cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57f770d65e0240418c10f071ae3bc1cf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39705 } }
I20260928 05:17:28.734450 3508 sys_catalog.cc:564] T 00000000000000000000000000000000 P 57f770d65e0240418c10f071ae3bc1cf [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:17:28.736173 3515 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57f770d65e0240418c10f071ae3bc1cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57f770d65e0240418c10f071ae3bc1cf. Latest consensus state: current_term: 1 leader_uuid: "57f770d65e0240418c10f071ae3bc1cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57f770d65e0240418c10f071ae3bc1cf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39705 } } }
I20260928 05:17:28.736438 3515 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57f770d65e0240418c10f071ae3bc1cf [sys.catalog]: This master's current role is: LEADER
I20260928 05:17:28.737221 3518 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:17:28.737110 3514 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57f770d65e0240418c10f071ae3bc1cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "57f770d65e0240418c10f071ae3bc1cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57f770d65e0240418c10f071ae3bc1cf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39705 } } }
I20260928 05:17:28.737416 3514 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57f770d65e0240418c10f071ae3bc1cf [sys.catalog]: This master's current role is: LEADER
I20260928 05:17:28.741510 3518 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:17:28.749176 3518 catalog_manager.cc:1347] Generated new cluster ID: 74c7f69a12f0430a892294b766719a25
I20260928 05:17:28.749272 3518 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:17:28.798919 3518 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:17:28.800220 3518 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:17:28.806064 3518 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 57f770d65e0240418c10f071ae3bc1cf: Generated new TSK 0
I20260928 05:17:28.806802 3518 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572622576160-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572622576160-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260928 05:17:28.849840 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572622576160-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572622576160-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572622576160-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572622576160-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39705
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572622576160-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:17:29.147360 3534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:17:29.147746 3534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:17:29.147894 3534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:17:29.155501 3534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:17:29.155591 3534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:17:29.155704 3534 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 05:17:29.164719 3534 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572622576160-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572622576160-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572622576160-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572622576160-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39705
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572622576160-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:17:29.165843 3534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:17:29.167038 3534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:17:29.178581 3542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:29.178561 3540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:29.178910 3539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:17:29.179165 3534 server_base.cc:1034] running on GCE node
I20260928 05:17:29.179692 3534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:17:29.180459 3534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:17:29.181633 3534 hybrid_clock.cc:648] HybridClock initialized: now 1790572649181581 us; error 43 us; skew 500 ppm
I20260928 05:17:29.181897 3534 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:17:29 dist-test-slave-whsn krb5kdc[3428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572649, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:17:29.184597 3534 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260928 05:17:29.185832 3534 webserver.cc:460] Webserver started at http://127.0.97.1:46685/ using document root <none> and password file <none>
I20260928 05:17:29.186441 3534 fs_manager.cc:359] Metadata directory not provided
I20260928 05:17:29.186548 3534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:17:29.186803 3534 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:17:29.189405 3534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572622576160-388-0/minicluster-data/ts-0/data/instance:
uuid: "5d9bcb27321b4cb89d6acad9fe40f196"
format_stamp: "Formatted at 2026-09-28 05:17:29 on dist-test-slave-whsn"
server_key: "fcce7ed8dd9253e47d7f7f9a5fc27f29"
server_key_iv: "fe4984a4f13d78285f6a3eb54a55fd70"
server_key_version: "encryptionkey@0"
I20260928 05:17:29.190106 3534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572622576160-388-0/minicluster-data/ts-0/wal/instance:
uuid: "5d9bcb27321b4cb89d6acad9fe40f196"
format_stamp: "Formatted at 2026-09-28 05:17:29 on dist-test-slave-whsn"
server_key: "fcce7ed8dd9253e47d7f7f9a5fc27f29"
server_key_iv: "fe4984a4f13d78285f6a3eb54a55fd70"
server_key_version: "encryptionkey@0"
I20260928 05:17:29.194506 3534 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 05:17:29.197469 3549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:29.198621 3534 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 05:17:29.198751 3534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572622576160-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572622576160-388-0/minicluster-data/ts-0/wal
uuid: "5d9bcb27321b4cb89d6acad9fe40f196"
format_stamp: "Formatted at 2026-09-28 05:17:29 on dist-test-slave-whsn"
server_key: "fcce7ed8dd9253e47d7f7f9a5fc27f29"
server_key_iv: "fe4984a4f13d78285f6a3eb54a55fd70"
server_key_version: "encryptionkey@0"
I20260928 05:17:29.198910 3534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572622576160-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572622576160-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572622576160-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:17:29.216566 3534 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:17:29.219803 3534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:17:29.220036 3534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:17:29.220803 3534 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:17:29.221951 3534 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:17:29.222023 3534 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:29.222137 3534 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:17:29.222186 3534 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:29.249102 3534 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37809
I20260928 05:17:29.249135 3661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37809 every 8 connection(s)
I20260928 05:17:29.250187 3534 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572622576160-388-0/minicluster-data/ts-0/data/info.pb
I20260928 05:17:29.252786 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 3534
I20260928 05:17:29.252982 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572622576160-388-0/minicluster-data/ts-0/wal/instance
I20260928 05:17:29.253536 388 external_mini_cluster.cc:1428] Setting key d6e454f2f7b879ce575555b075e85503
Sep 28 05:17:29 dist-test-slave-whsn krb5kdc[3428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572649, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:17:29.268949 3664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:29.252303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39945 (local address 127.0.97.62:39705)
0928 05:17:29.252724 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:29.252729 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:29.253712 (+ 983us) server_negotiation.cc:407] Connection header received
0928 05:17:29.255521 (+ 1809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:29.255526 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:29.255599 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:29.255714 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:29.257000 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:29.257955 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:29.259337 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:29.259849 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:29.259940 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:29.263232 (+ 3292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:29.263238 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:29.263242 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:29.263277 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:29.266174 (+ 2897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:29.266832 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:29.266837 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:29.266841 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:29.266981 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:29.267333 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:29.267338 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:29.267341 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:29.267681 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:29.267828 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:29.268356 (+ 528us) server_negotiation.cc:299] Negotiation successful
0928 05:17:29.268567 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":199,"threads_started":1}
I20260928 05:17:29.269523 3665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:29.252799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39705 (local address 127.0.97.1:39945)
0928 05:17:29.253547 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:29.253605 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:29.255291 (+ 1686us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:29.255871 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:29.255878 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:29.256313 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:17:29.256821 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:29.256837 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:29.258073 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:29.258076 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:29.259220 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:29.259229 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:29.260052 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:29.260058 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:29.260322 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:29.260947 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:17:29.260972 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:17:29.263020 (+ 2048us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:17:29.266308 (+ 3288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:29.266316 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:29.266327 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:29.266692 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:29.267071 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:29.267074 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:29.267076 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:29.267232 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:29.267784 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:17:29.267789 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:17:29.268037 (+ 248us) client_negotiation.cc:769] Sending connection context
0928 05:17:29.268234 (+ 197us) client_negotiation.cc:240] Negotiation successful
0928 05:17:29.268546 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":295,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260928 05:17:29.270831 3662 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39705
I20260928 05:17:29.271092 3662 heartbeater.cc:464] Registering TS with master...
I20260928 05:17:29.271703 3662 heartbeater.cc:511] Master 127.0.97.62:39705 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260928 05:17:29.273563 3473 ts_manager.cc:194] Registered new tserver with Master: 5d9bcb27321b4cb89d6acad9fe40f196 (127.0.97.1:37809)
I20260928 05:17:29.274906 3473 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:39945
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572622576160-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572622576160-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260928 05:17:29.289691 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572622576160-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572622576160-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572622576160-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572622576160-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39705
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572622576160-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:17:29.580703 3668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:17:29.580965 3668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:17:29.581097 3668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:17:29.587992 3668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:17:29.588083 3668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:17:29.588217 3668 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 05:17:29.597132 3668 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572622576160-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572622576160-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572622576160-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572622576160-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39705
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572622576160-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:17:29.598470 3668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:17:29.600097 3668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:17:29.614809 3674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:29.615358 3676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:29.615388 3673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:17:29.615839 3668 server_base.cc:1034] running on GCE node
I20260928 05:17:29.616442 3668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:17:29.617326 3668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:17:29.618503 3668 hybrid_clock.cc:648] HybridClock initialized: now 1790572649618441 us; error 59 us; skew 500 ppm
I20260928 05:17:29.618790 3668 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:17:29 dist-test-slave-whsn krb5kdc[3428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572649, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:17:29.622051 3668 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260928 05:17:29.623395 3668 webserver.cc:460] Webserver started at http://127.0.97.2:40899/ using document root <none> and password file <none>
I20260928 05:17:29.624013 3668 fs_manager.cc:359] Metadata directory not provided
I20260928 05:17:29.624109 3668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:17:29.624364 3668 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:17:29.627817 3668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572622576160-388-0/minicluster-data/ts-1/data/instance:
uuid: "5d47a2553beb476d88332a0a200568c4"
format_stamp: "Formatted at 2026-09-28 05:17:29 on dist-test-slave-whsn"
server_key: "6ebf71043b9045a64f3de668992c96e5"
server_key_iv: "aea3e16f71fefd0205375ef609957a96"
server_key_version: "encryptionkey@0"
I20260928 05:17:29.628713 3668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572622576160-388-0/minicluster-data/ts-1/wal/instance:
uuid: "5d47a2553beb476d88332a0a200568c4"
format_stamp: "Formatted at 2026-09-28 05:17:29 on dist-test-slave-whsn"
server_key: "6ebf71043b9045a64f3de668992c96e5"
server_key_iv: "aea3e16f71fefd0205375ef609957a96"
server_key_version: "encryptionkey@0"
I20260928 05:17:29.633741 3668 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 05:17:29.636842 3683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:29.638052 3668 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 05:17:29.638182 3668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572622576160-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572622576160-388-0/minicluster-data/ts-1/wal
uuid: "5d47a2553beb476d88332a0a200568c4"
format_stamp: "Formatted at 2026-09-28 05:17:29 on dist-test-slave-whsn"
server_key: "6ebf71043b9045a64f3de668992c96e5"
server_key_iv: "aea3e16f71fefd0205375ef609957a96"
server_key_version: "encryptionkey@0"
I20260928 05:17:29.638347 3668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572622576160-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572622576160-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572622576160-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:17:29.656540 3668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:17:29.660244 3668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:17:29.660498 3668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:17:29.661352 3668 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:17:29.662530 3668 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:17:29.662598 3668 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:29.662693 3668 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:17:29.662739 3668 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:29.691002 3668 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43715
I20260928 05:17:29.691068 3795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43715 every 8 connection(s)
I20260928 05:17:29.692121 3668 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572622576160-388-0/minicluster-data/ts-1/data/info.pb
I20260928 05:17:29.701749 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 3668
I20260928 05:17:29.701934 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572622576160-388-0/minicluster-data/ts-1/wal/instance
I20260928 05:17:29.702368 388 external_mini_cluster.cc:1428] Setting key 44955b2e11ba6f8c6517cc42b306bccf
Sep 28 05:17:29 dist-test-slave-whsn krb5kdc[3428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572649, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:17:29.713469 3664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:29.694352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36731 (local address 127.0.97.62:39705)
0928 05:17:29.694536 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:29.694542 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:29.695704 (+ 1162us) server_negotiation.cc:407] Connection header received
0928 05:17:29.698055 (+ 2351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:29.698059 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:29.698139 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:29.698223 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:29.699940 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:29.700678 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:29.702913 (+ 2235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:29.703583 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:29.703671 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:29.707836 (+ 4165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:29.707842 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:29.707847 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:29.707884 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:29.710777 (+ 2893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:29.711420 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:29.711425 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:29.711429 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:29.711564 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:29.711942 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:29.711946 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:29.711948 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:29.712179 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:29.712300 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:29.712947 (+ 647us) server_negotiation.cc:299] Negotiation successful
0928 05:17:29.713152 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 05:17:29.714337 3798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:29.694853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39705 (local address 127.0.97.2:36731)
0928 05:17:29.695527 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:29.695575 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:29.697771 (+ 2196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:29.698454 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:29.698465 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:29.699105 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:17:29.699800 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:29.699820 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:29.700815 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:29.700821 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:29.702754 (+ 1933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:29.702767 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:29.703955 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:29.703962 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:29.704220 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:29.704972 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:17:29.705003 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:17:29.707630 (+ 2627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:17:29.710899 (+ 3269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:29.710908 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:29.710923 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:29.711301 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:29.711660 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:29.711665 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:29.711669 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:29.711833 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:29.712291 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:17:29.712298 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:17:29.712588 (+ 290us) client_negotiation.cc:769] Sending connection context
0928 05:17:29.712806 (+ 218us) client_negotiation.cc:240] Negotiation successful
0928 05:17:29.713306 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":265,"threads_started":1}
I20260928 05:17:29.715636 3796 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39705
I20260928 05:17:29.715935 3796 heartbeater.cc:464] Registering TS with master...
I20260928 05:17:29.716745 3796 heartbeater.cc:511] Master 127.0.97.62:39705 requested a full tablet report, sending...
I20260928 05:17:29.718132 3473 ts_manager.cc:194] Registered new tserver with Master: 5d47a2553beb476d88332a0a200568c4 (127.0.97.2:43715)
I20260928 05:17:29.718871 3473 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:36731
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572622576160-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572622576160-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260928 05:17:29.739878 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572622576160-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572622576160-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572622576160-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572622576160-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39705
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572622576160-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:17:30.031420 3801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:17:30.031659 3801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:17:30.031754 3801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:17:30.038619 3801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:17:30.038705 3801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:17:30.038815 3801 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 05:17:30.047820 3801 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572622576160-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572622576160-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572622576160-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572622576160-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39705
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572622576160-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:17:30.048884 3801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:17:30.050050 3801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:17:30.061510 3806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:30.061488 3807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:30.061446 3809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:17:30.062806 3801 server_base.cc:1034] running on GCE node
I20260928 05:17:30.063311 3801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:17:30.064105 3801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:17:30.065281 3801 hybrid_clock.cc:648] HybridClock initialized: now 1790572650065238 us; error 39 us; skew 500 ppm
I20260928 05:17:30.065568 3801 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:17:30 dist-test-slave-whsn krb5kdc[3428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572650, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:17:30.068303 3801 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260928 05:17:30.069563 3801 webserver.cc:460] Webserver started at http://127.0.97.3:45767/ using document root <none> and password file <none>
I20260928 05:17:30.070169 3801 fs_manager.cc:359] Metadata directory not provided
I20260928 05:17:30.070271 3801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:17:30.070529 3801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:17:30.073186 3801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572622576160-388-0/minicluster-data/ts-2/data/instance:
uuid: "2cc9ce4ca11a42c0b98561ea8efbea2e"
format_stamp: "Formatted at 2026-09-28 05:17:30 on dist-test-slave-whsn"
server_key: "7287fc993a09c44b266de9697b99875b"
server_key_iv: "f9bd2365e73b257b892c44baefc301ba"
server_key_version: "encryptionkey@0"
I20260928 05:17:30.073863 3801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572622576160-388-0/minicluster-data/ts-2/wal/instance:
uuid: "2cc9ce4ca11a42c0b98561ea8efbea2e"
format_stamp: "Formatted at 2026-09-28 05:17:30 on dist-test-slave-whsn"
server_key: "7287fc993a09c44b266de9697b99875b"
server_key_iv: "f9bd2365e73b257b892c44baefc301ba"
server_key_version: "encryptionkey@0"
I20260928 05:17:30.078301 3801 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 05:17:30.081266 3816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:30.082432 3801 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260928 05:17:30.082562 3801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572622576160-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572622576160-388-0/minicluster-data/ts-2/wal
uuid: "2cc9ce4ca11a42c0b98561ea8efbea2e"
format_stamp: "Formatted at 2026-09-28 05:17:30 on dist-test-slave-whsn"
server_key: "7287fc993a09c44b266de9697b99875b"
server_key_iv: "f9bd2365e73b257b892c44baefc301ba"
server_key_version: "encryptionkey@0"
I20260928 05:17:30.082711 3801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572622576160-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572622576160-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572622576160-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:17:30.111840 3801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:17:30.115291 3801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:17:30.115542 3801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:17:30.116369 3801 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:17:30.117574 3801 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:17:30.117645 3801 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:30.117766 3801 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:17:30.117817 3801 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:30.145797 3801 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46533
I20260928 05:17:30.145839 3928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46533 every 8 connection(s)
I20260928 05:17:30.147166 3801 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572622576160-388-0/minicluster-data/ts-2/data/info.pb
I20260928 05:17:30.152519 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 3801
I20260928 05:17:30.152688 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572622576160-388-0/minicluster-data/ts-2/wal/instance
I20260928 05:17:30.153244 388 external_mini_cluster.cc:1428] Setting key 58add6b31023ee610c47c34351b3ad71
Sep 28 05:17:30 dist-test-slave-whsn krb5kdc[3428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572650, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:17:30.165868 3664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:30.149382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55731 (local address 127.0.97.62:39705)
0928 05:17:30.149537 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:30.149542 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:30.150720 (+ 1178us) server_negotiation.cc:407] Connection header received
0928 05:17:30.153025 (+ 2305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:30.153031 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:30.153114 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:30.153214 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:30.155018 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:30.155783 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:30.157332 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:30.157731 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:30.157833 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:30.160841 (+ 3008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:30.160846 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:30.160867 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:30.160913 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:30.163220 (+ 2307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:30.163826 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:30.163832 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:30.163836 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:30.163949 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:30.164305 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:30.164310 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:30.164314 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:30.164555 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:30.164688 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:30.165351 (+ 663us) server_negotiation.cc:299] Negotiation successful
0928 05:17:30.165544 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 05:17:30.166801 3931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:30.149840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39705 (local address 127.0.97.3:55731)
0928 05:17:30.150571 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:30.150619 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:30.152673 (+ 2054us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:30.153617 (+ 944us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:30.153628 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:30.154138 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:17:30.154858 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:30.154881 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:30.155917 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:30.155922 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:30.157208 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:30.157220 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:30.157931 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:30.157938 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:30.158164 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:30.158723 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:17:30.158743 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:17:30.160654 (+ 1911us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:17:30.163335 (+ 2681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:30.163344 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:30.163358 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:30.163723 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:30.164042 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:30.164046 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:30.164050 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:30.164210 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:30.164679 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:17:30.164686 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:17:30.165010 (+ 324us) client_negotiation.cc:769] Sending connection context
0928 05:17:30.165250 (+ 240us) client_negotiation.cc:240] Negotiation successful
0928 05:17:30.165695 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":286,"threads_started":1}
I20260928 05:17:30.167972 3929 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39705
I20260928 05:17:30.168210 3929 heartbeater.cc:464] Registering TS with master...
I20260928 05:17:30.168835 3929 heartbeater.cc:511] Master 127.0.97.62:39705 requested a full tablet report, sending...
I20260928 05:17:30.170099 3472 ts_manager.cc:194] Registered new tserver with Master: 2cc9ce4ca11a42c0b98561ea8efbea2e (127.0.97.3:46533)
I20260928 05:17:30.170874 3472 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:55731
I20260928 05:17:30.179741 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 05:17:30 dist-test-slave-whsn krb5kdc[3428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572650, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 28 05:17:30 dist-test-slave-whsn krb5kdc[3428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572650, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:17:30.220985 3664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:30.208375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47464 (local address 127.0.97.62:39705)
0928 05:17:30.208532 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:30.208537 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:30.209014 (+ 477us) server_negotiation.cc:407] Connection header received
0928 05:17:30.209179 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:30.209183 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:30.209246 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:30.209319 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:30.210540 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:30.211235 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:30.212462 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:30.212888 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:30.212991 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:30.216540 (+ 3549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:30.216544 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:30.216548 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:30.216581 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:30.218811 (+ 2230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:30.219386 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:30.219390 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:30.219392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:30.219506 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:30.219769 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:30.219771 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:30.219773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:30.220110 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:30.220274 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:30.220551 (+ 277us) server_negotiation.cc:299] Negotiation successful
0928 05:17:30.220703 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260928 05:17:30.221468 3472 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:47464
I20260928 05:17:30.223443 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 3534
Sep 28 05:17:30 dist-test-slave-whsn krb5kdc[3428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572650, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260928 05:17:30.237172 3940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:30.224213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59762 (local address 127.0.97.1:37809)
0928 05:17:30.224737 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:30.224743 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:30.224767 (+ 24us) server_negotiation.cc:407] Connection header received
0928 05:17:30.224816 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:30.224821 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:30.225073 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:30.225200 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:30.226135 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:30.227077 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:30.228159 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:30.228711 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:30.228808 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:30.232183 (+ 3375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:30.232194 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:30.232207 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:30.232240 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:30.234693 (+ 2453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:30.235299 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:30.235306 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:30.235308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:30.235422 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:30.235753 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:30.235757 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:30.235759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:30.236069 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:30.236341 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:30.236601 (+ 260us) server_negotiation.cc:299] Negotiation successful
0928 05:17:30.236842 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":268,"threads_started":1}
W20260928 05:17:30.238317 3657 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:59762
W20260928 05:17:30.239423 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 3534: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 05:17:30.239586 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 3534
I20260928 05:17:30.251703 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 3668
Sep 28 05:17:30 dist-test-slave-whsn krb5kdc[3428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572650, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260928 05:17:30.265904 3941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:30.252454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58234 (local address 127.0.97.2:43715)
0928 05:17:30.252987 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:30.252994 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:30.253018 (+ 24us) server_negotiation.cc:407] Connection header received
0928 05:17:30.253088 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:30.253095 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:30.253293 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:30.253431 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:30.254477 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:30.255512 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:30.256673 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:30.257252 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:30.257349 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:30.260954 (+ 3605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:30.260964 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:30.260978 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:30.261016 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:30.263398 (+ 2382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:30.263970 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:30.263978 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:30.263980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:30.264113 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:30.264408 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:30.264411 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:30.264413 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:30.264732 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:30.265041 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:30.265344 (+ 303us) server_negotiation.cc:299] Negotiation successful
0928 05:17:30.265600 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":250,"threads_started":1}
W20260928 05:17:30.267110 3791 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:58234
W20260928 05:17:30.268076 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 3668: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 05:17:30.268229 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 3668
I20260928 05:17:30.279881 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 3801
Sep 28 05:17:30 dist-test-slave-whsn krb5kdc[3428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572650, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260928 05:17:30.294699 3942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:30.280689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49336 (local address 127.0.97.3:46533)
0928 05:17:30.281299 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:30.281305 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:30.281332 (+ 27us) server_negotiation.cc:407] Connection header received
0928 05:17:30.281385 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:30.281390 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:30.281585 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:30.281726 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:30.282889 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:30.283942 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:30.285186 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:30.285757 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:30.285869 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:30.289607 (+ 3738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:30.289616 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:30.289630 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:30.289660 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:30.292170 (+ 2510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:30.292742 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:30.292749 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:30.292751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:30.292952 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:30.293292 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:30.293297 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:30.293300 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:30.293604 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:30.293878 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:30.294149 (+ 271us) server_negotiation.cc:299] Negotiation successful
0928 05:17:30.294384 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":330,"threads_started":1}
W20260928 05:17:30.295868 3924 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:49336
W20260928 05:17:30.296921 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 3801: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 05:17:30.297049 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 3801
I20260928 05:17:30.308611 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 3441
I20260928 05:17:30.309612 3503 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:17:30.407696 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 3441
2026-09-28T05:17:30Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3242 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572622576160-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:17:30 dist-test-slave-whsn krb5kdc[3946](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:17:30 dist-test-slave-whsn krb5kdc[3946](info): set up 2 sockets
Sep 28 05:17:30 dist-test-slave-whsn krb5kdc[3946](info): commencing operation
krb5kdc: starting...
W20260928 05:17:31.396785 388 mini_kdc.cc:121] Time spent starting KDC: real 0.953s user 0.004s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572622576160-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572622576160-388-0/krb5kdc/test-user.keytab.
Sep 28 05:17:31 dist-test-slave-whsn krb5kdc[3946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:17:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:17:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 05:17:31.558281 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572622576160-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572622576160-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572622576160-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45007
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45007
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:17:31.848889 3959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:17:31.849151 3959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:17:31.849278 3959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:17:31.856117 3959 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:17:31.856205 3959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:17:31.856272 3959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:17:31.856346 3959 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:17:31.856405 3959 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:17:31.865727 3959 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572622576160-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572622576160-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45007
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572622576160-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:17:31.866894 3959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:17:31.868093 3959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:17:31.878580 3964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:31.878533 3965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:31.878602 3967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:17:31.879555 3959 server_base.cc:1034] running on GCE node
I20260928 05:17:31.880213 3959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:17:31.881445 3959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:17:31.882637 3959 hybrid_clock.cc:648] HybridClock initialized: now 1790572651882597 us; error 25 us; skew 500 ppm
I20260928 05:17:31.882932 3959 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:17:31 dist-test-slave-whsn krb5kdc[3946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572651, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:17:31.885664 3959 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 05:17:31.886965 3959 webserver.cc:460] Webserver started at http://127.0.97.62:33621/ using document root <none> and password file <none>
I20260928 05:17:31.887565 3959 fs_manager.cc:359] Metadata directory not provided
I20260928 05:17:31.887667 3959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:17:31.887940 3959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:17:31.890677 3959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572622576160-388-0/minicluster-data/master-0/data/instance:
uuid: "649de44fecef4586a8e51ebb33728c6c"
format_stamp: "Formatted at 2026-09-28 05:17:31 on dist-test-slave-whsn"
server_key: "224a7bb69aee53fc2a5e57ecd053e29f"
server_key_iv: "1212ae40596c445a737a45ef9aec569c"
server_key_version: "encryptionkey@0"
I20260928 05:17:31.891366 3959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572622576160-388-0/minicluster-data/master-0/wal/instance:
uuid: "649de44fecef4586a8e51ebb33728c6c"
format_stamp: "Formatted at 2026-09-28 05:17:31 on dist-test-slave-whsn"
server_key: "224a7bb69aee53fc2a5e57ecd053e29f"
server_key_iv: "1212ae40596c445a737a45ef9aec569c"
server_key_version: "encryptionkey@0"
I20260928 05:17:31.895892 3959 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260928 05:17:31.898942 3974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:31.900070 3959 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 05:17:31.900208 3959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572622576160-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572622576160-388-0/minicluster-data/master-0/wal
uuid: "649de44fecef4586a8e51ebb33728c6c"
format_stamp: "Formatted at 2026-09-28 05:17:31 on dist-test-slave-whsn"
server_key: "224a7bb69aee53fc2a5e57ecd053e29f"
server_key_iv: "1212ae40596c445a737a45ef9aec569c"
server_key_version: "encryptionkey@0"
I20260928 05:17:31.900368 3959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572622576160-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572622576160-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572622576160-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:17:31.913530 3959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:17:31.916987 3959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:17:31.917217 3959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:17:31.932358 3959 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45007
I20260928 05:17:31.932389 4025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45007 every 8 connection(s)
I20260928 05:17:31.933568 3959 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:17:31.936808 4026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:17:31.941363 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 3959
I20260928 05:17:31.941558 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572622576160-388-0/minicluster-data/master-0/wal/instance
I20260928 05:17:31.942277 388 external_mini_cluster.cc:1428] Setting key 0860519cb0c479d600747dc6fa79c8b5
I20260928 05:17:31.947099 4026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 649de44fecef4586a8e51ebb33728c6c: Bootstrap starting.
I20260928 05:17:31.950254 4026 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 649de44fecef4586a8e51ebb33728c6c: Neither blocks nor log segments found. Creating new log.
I20260928 05:17:31.951359 4026 log.cc:824] T 00000000000000000000000000000000 P 649de44fecef4586a8e51ebb33728c6c: Log is configured to *not* fsync() on all Append() calls
I20260928 05:17:31.954268 4026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 649de44fecef4586a8e51ebb33728c6c: No bootstrap required, opened a new log
Sep 28 05:17:31 dist-test-slave-whsn krb5kdc[3946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:17:31.962298 4026 raft_consensus.cc:348] T 00000000000000000000000000000000 P 649de44fecef4586a8e51ebb33728c6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "649de44fecef4586a8e51ebb33728c6c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45007 } }
I20260928 05:17:31.962621 4026 raft_consensus.cc:374] T 00000000000000000000000000000000 P 649de44fecef4586a8e51ebb33728c6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:17:31.962726 4026 raft_consensus.cc:729] T 00000000000000000000000000000000 P 649de44fecef4586a8e51ebb33728c6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 649de44fecef4586a8e51ebb33728c6c, State: Initialized, Role: FOLLOWER
I20260928 05:17:31.963294 4029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:31.944079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51664 (local address 127.0.97.62:45007)
0928 05:17:31.944782 (+ 703us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:31.944793 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:31.944839 (+ 46us) server_negotiation.cc:407] Connection header received
0928 05:17:31.946459 (+ 1620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:31.946479 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:31.946830 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:31.947109 (+ 279us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:31.948707 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:31.949945 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:31.951607 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:31.952556 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:31.952670 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:31.956901 (+ 4231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:31.956909 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:31.956922 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:31.956955 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:31.959681 (+ 2726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:31.960364 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:31.960372 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:31.960377 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:31.960538 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:31.960904 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:31.960907 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:31.960909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:31.961375 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:31.961579 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:31.961927 (+ 348us) server_negotiation.cc:299] Negotiation successful
0928 05:17:31.962220 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":314,"threads_started":1}
I20260928 05:17:31.963219 4026 consensus_queue.cc:260] T 00000000000000000000000000000000 P 649de44fecef4586a8e51ebb33728c6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "649de44fecef4586a8e51ebb33728c6c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45007 } }
I20260928 05:17:31.963419 4026 raft_consensus.cc:388] T 00000000000000000000000000000000 P 649de44fecef4586a8e51ebb33728c6c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:17:31.963538 4026 raft_consensus.cc:482] T 00000000000000000000000000000000 P 649de44fecef4586a8e51ebb33728c6c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:17:31.963690 4026 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 649de44fecef4586a8e51ebb33728c6c [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:17:31.965755 4026 raft_consensus.cc:504] T 00000000000000000000000000000000 P 649de44fecef4586a8e51ebb33728c6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "649de44fecef4586a8e51ebb33728c6c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45007 } }
I20260928 05:17:31.966161 4026 leader_election.cc:302] T 00000000000000000000000000000000 P 649de44fecef4586a8e51ebb33728c6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 649de44fecef4586a8e51ebb33728c6c; no voters:
I20260928 05:17:31.966634 4026 leader_election.cc:288] T 00000000000000000000000000000000 P 649de44fecef4586a8e51ebb33728c6c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:17:31.966732 4031 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 649de44fecef4586a8e51ebb33728c6c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:17:31.967334 4031 raft_consensus.cc:686] T 00000000000000000000000000000000 P 649de44fecef4586a8e51ebb33728c6c [term 1 LEADER]: Becoming Leader. State: Replica: 649de44fecef4586a8e51ebb33728c6c, State: Running, Role: LEADER
I20260928 05:17:31.967801 4031 consensus_queue.cc:237] T 00000000000000000000000000000000 P 649de44fecef4586a8e51ebb33728c6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "649de44fecef4586a8e51ebb33728c6c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45007 } }
I20260928 05:17:31.968267 4026 sys_catalog.cc:564] T 00000000000000000000000000000000 P 649de44fecef4586a8e51ebb33728c6c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:17:31.970358 4032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 649de44fecef4586a8e51ebb33728c6c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "649de44fecef4586a8e51ebb33728c6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "649de44fecef4586a8e51ebb33728c6c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45007 } } }
I20260928 05:17:31.970427 4033 sys_catalog.cc:455] T 00000000000000000000000000000000 P 649de44fecef4586a8e51ebb33728c6c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 649de44fecef4586a8e51ebb33728c6c. Latest consensus state: current_term: 1 leader_uuid: "649de44fecef4586a8e51ebb33728c6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "649de44fecef4586a8e51ebb33728c6c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45007 } } }
I20260928 05:17:31.970762 4032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 649de44fecef4586a8e51ebb33728c6c [sys.catalog]: This master's current role is: LEADER
I20260928 05:17:31.970798 4033 sys_catalog.cc:458] T 00000000000000000000000000000000 P 649de44fecef4586a8e51ebb33728c6c [sys.catalog]: This master's current role is: LEADER
I20260928 05:17:31.971526 4040 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:17:31.978042 4040 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:17:31.986395 4040 catalog_manager.cc:1347] Generated new cluster ID: 79807d9466d447b9a7f3d64c6dd09d1a
I20260928 05:17:31.986503 4040 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:17:32.009752 4040 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:17:32.010927 4040 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:17:32.016606 4040 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 649de44fecef4586a8e51ebb33728c6c: Generated new TSK 0
I20260928 05:17:32.017360 4040 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572622576160-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572622576160-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260928 05:17:32.064229 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572622576160-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572622576160-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572622576160-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572622576160-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45007
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572622576160-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:17:32.360405 4052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:17:32.360635 4052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:17:32.360743 4052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:17:32.367661 4052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:17:32.367753 4052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:17:32.367923 4052 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 05:17:32.377035 4052 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572622576160-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572622576160-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572622576160-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572622576160-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45007
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572622576160-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:17:32.378183 4052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:17:32.379415 4052 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:17:32.391667 4058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:32.391611 4060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:32.392186 4057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:17:32.392426 4052 server_base.cc:1034] running on GCE node
I20260928 05:17:32.393046 4052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:17:32.393882 4052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:17:32.395081 4052 hybrid_clock.cc:648] HybridClock initialized: now 1790572652395026 us; error 53 us; skew 500 ppm
I20260928 05:17:32.395365 4052 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:17:32 dist-test-slave-whsn krb5kdc[3946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572652, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:17:32.401855 4052 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260928 05:17:32.403167 4052 webserver.cc:460] Webserver started at http://127.0.97.1:38389/ using document root <none> and password file <none>
I20260928 05:17:32.403805 4052 fs_manager.cc:359] Metadata directory not provided
I20260928 05:17:32.403913 4052 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:17:32.404206 4052 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:17:32.406946 4052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572622576160-388-0/minicluster-data/ts-0/data/instance:
uuid: "d831f16c38c1454cbdc4314bef1dabd0"
format_stamp: "Formatted at 2026-09-28 05:17:32 on dist-test-slave-whsn"
server_key: "c362f617492d7aa725ce3e95118baa94"
server_key_iv: "7aeb094a375dc18ea42b1d59ac3b7760"
server_key_version: "encryptionkey@0"
I20260928 05:17:32.407662 4052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572622576160-388-0/minicluster-data/ts-0/wal/instance:
uuid: "d831f16c38c1454cbdc4314bef1dabd0"
format_stamp: "Formatted at 2026-09-28 05:17:32 on dist-test-slave-whsn"
server_key: "c362f617492d7aa725ce3e95118baa94"
server_key_iv: "7aeb094a375dc18ea42b1d59ac3b7760"
server_key_version: "encryptionkey@0"
I20260928 05:17:32.412221 4052 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260928 05:17:32.415337 4067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:32.416571 4052 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 05:17:32.416708 4052 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572622576160-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572622576160-388-0/minicluster-data/ts-0/wal
uuid: "d831f16c38c1454cbdc4314bef1dabd0"
format_stamp: "Formatted at 2026-09-28 05:17:32 on dist-test-slave-whsn"
server_key: "c362f617492d7aa725ce3e95118baa94"
server_key_iv: "7aeb094a375dc18ea42b1d59ac3b7760"
server_key_version: "encryptionkey@0"
I20260928 05:17:32.416914 4052 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572622576160-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572622576160-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572622576160-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:17:32.438130 4052 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:17:32.441886 4052 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:17:32.442131 4052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:17:32.442978 4052 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:17:32.444156 4052 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:17:32.444231 4052 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:32.444344 4052 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:17:32.444394 4052 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:32.472326 4052 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44109
I20260928 05:17:32.472394 4179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44109 every 8 connection(s)
I20260928 05:17:32.473489 4052 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572622576160-388-0/minicluster-data/ts-0/data/info.pb
I20260928 05:17:32.478068 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 4052
I20260928 05:17:32.478240 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572622576160-388-0/minicluster-data/ts-0/wal/instance
I20260928 05:17:32.478791 388 external_mini_cluster.cc:1428] Setting key e948dc3d6307508d0fe414bf3ba180be
Sep 28 05:17:32 dist-test-slave-whsn krb5kdc[3946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572652, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:17:32.492004 4182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:32.475668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39059 (local address 127.0.97.62:45007)
0928 05:17:32.476025 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:32.476029 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:32.476921 (+ 892us) server_negotiation.cc:407] Connection header received
0928 05:17:32.478701 (+ 1780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:32.478707 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:32.478785 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:32.478877 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:32.480554 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:32.481572 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:32.482994 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:32.483512 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:32.483614 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:32.486853 (+ 3239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:32.486857 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:32.486861 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:32.486890 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:32.489280 (+ 2390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:32.490034 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:32.490038 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:32.490040 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:32.490147 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:32.490525 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:32.490528 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:32.490530 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:32.490753 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:32.490854 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:32.491559 (+ 705us) server_negotiation.cc:299] Negotiation successful
0928 05:17:32.491756 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":180,"threads_started":1}
I20260928 05:17:32.493288 4183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:32.476120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45007 (local address 127.0.97.1:39059)
0928 05:17:32.476776 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:32.476811 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:32.478454 (+ 1643us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:32.479105 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:32.479113 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:32.479689 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:17:32.480386 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:32.480412 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:32.481704 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:32.481708 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:32.482868 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:32.482879 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:32.483688 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:32.483692 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:32.483865 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:32.484445 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:17:32.484469 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:17:32.486666 (+ 2197us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:17:32.489398 (+ 2732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:32.489406 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:32.489420 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:32.489938 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:32.490233 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:32.490237 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:32.490240 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:32.490441 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:32.490857 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:17:32.490864 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:17:32.491211 (+ 347us) client_negotiation.cc:769] Sending connection context
0928 05:17:32.491480 (+ 269us) client_negotiation.cc:240] Negotiation successful
0928 05:17:32.491953 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":262,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260928 05:17:32.496821 4180 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45007
I20260928 05:17:32.497161 4180 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260928 05:17:32.497910 4180 heartbeater.cc:511] Master 127.0.97.62:45007 requested a full tablet report, sending...
I20260928 05:17:32.500182 3991 ts_manager.cc:194] Registered new tserver with Master: d831f16c38c1454cbdc4314bef1dabd0 (127.0.97.1:44109)
I20260928 05:17:32.502041 3991 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:39059
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572622576160-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572622576160-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260928 05:17:32.518098 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572622576160-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572622576160-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572622576160-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572622576160-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45007
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572622576160-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:17:32.810606 4186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:17:32.810839 4186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:17:32.810961 4186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:17:32.817829 4186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:17:32.817922 4186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:17:32.818063 4186 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 05:17:32.827132 4186 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572622576160-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572622576160-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572622576160-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572622576160-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45007
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572622576160-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:17:32.828213 4186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:17:32.829417 4186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:17:32.840811 4192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:32.840871 4191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:32.840910 4194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:17:32.841862 4186 server_base.cc:1034] running on GCE node
I20260928 05:17:32.842397 4186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:17:32.843178 4186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:17:32.844369 4186 hybrid_clock.cc:648] HybridClock initialized: now 1790572652844320 us; error 46 us; skew 500 ppm
I20260928 05:17:32.844655 4186 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:17:32 dist-test-slave-whsn krb5kdc[3946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572652, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:17:32.847402 4186 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260928 05:17:32.848639 4186 webserver.cc:460] Webserver started at http://127.0.97.2:35017/ using document root <none> and password file <none>
I20260928 05:17:32.849263 4186 fs_manager.cc:359] Metadata directory not provided
I20260928 05:17:32.849356 4186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:17:32.849586 4186 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:17:32.852105 4186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572622576160-388-0/minicluster-data/ts-1/data/instance:
uuid: "5db6e0417ab64f7f86d8e9a68430fa2b"
format_stamp: "Formatted at 2026-09-28 05:17:32 on dist-test-slave-whsn"
server_key: "1c9dd5ec43b8cef46f74a41f2c21d403"
server_key_iv: "20ba2d93219ace7f53e966e68f66a7f9"
server_key_version: "encryptionkey@0"
I20260928 05:17:32.852766 4186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572622576160-388-0/minicluster-data/ts-1/wal/instance:
uuid: "5db6e0417ab64f7f86d8e9a68430fa2b"
format_stamp: "Formatted at 2026-09-28 05:17:32 on dist-test-slave-whsn"
server_key: "1c9dd5ec43b8cef46f74a41f2c21d403"
server_key_iv: "20ba2d93219ace7f53e966e68f66a7f9"
server_key_version: "encryptionkey@0"
I20260928 05:17:32.857198 4186 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 05:17:32.860194 4201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:32.861438 4186 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 05:17:32.861554 4186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572622576160-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572622576160-388-0/minicluster-data/ts-1/wal
uuid: "5db6e0417ab64f7f86d8e9a68430fa2b"
format_stamp: "Formatted at 2026-09-28 05:17:32 on dist-test-slave-whsn"
server_key: "1c9dd5ec43b8cef46f74a41f2c21d403"
server_key_iv: "20ba2d93219ace7f53e966e68f66a7f9"
server_key_version: "encryptionkey@0"
I20260928 05:17:32.861701 4186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572622576160-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572622576160-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572622576160-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:17:32.884130 4186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:17:32.887727 4186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:17:32.887948 4186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:17:32.888803 4186 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:17:32.890031 4186 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:17:32.890100 4186 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:32.890183 4186 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:17:32.890223 4186 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:32.918463 4186 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43625
I20260928 05:17:32.918525 4313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43625 every 8 connection(s)
I20260928 05:17:32.919653 4186 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572622576160-388-0/minicluster-data/ts-1/data/info.pb
I20260928 05:17:32.921487 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 4186
I20260928 05:17:32.921654 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572622576160-388-0/minicluster-data/ts-1/wal/instance
I20260928 05:17:32.922119 388 external_mini_cluster.cc:1428] Setting key 36b7ffc66992e4de455e8e35060bfe29
Sep 28 05:17:32 dist-test-slave-whsn krb5kdc[3946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572652, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:17:32.939101 4182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:32.922175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41195 (local address 127.0.97.62:45007)
0928 05:17:32.922359 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:32.922365 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:32.923485 (+ 1120us) server_negotiation.cc:407] Connection header received
0928 05:17:32.925830 (+ 2345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:32.925835 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:32.925895 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:32.925968 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:32.927724 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:32.928369 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:32.930228 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:32.930834 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:32.930952 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:32.933997 (+ 3045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:32.934002 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:32.934005 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:32.934037 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:32.936370 (+ 2333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:32.937045 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:32.937051 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:32.937055 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:32.937195 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:32.937574 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:32.937579 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:32.937583 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:32.937828 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:32.937961 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:32.938549 (+ 588us) server_negotiation.cc:299] Negotiation successful
0928 05:17:32.938749 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 05:17:32.939963 4316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:32.922662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45007 (local address 127.0.97.2:41195)
0928 05:17:32.923347 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:32.923381 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:32.925545 (+ 2164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:32.926220 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:32.926231 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:32.926862 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:17:32.927577 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:32.927602 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:32.928526 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:32.928531 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:32.930095 (+ 1564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:32.930107 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:32.931002 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:32.931006 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:32.931186 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:32.931769 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:17:32.931790 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:17:32.933803 (+ 2013us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:17:32.936525 (+ 2722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:32.936530 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:32.936541 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:32.936933 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:32.937291 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:32.937296 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:32.937299 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:32.937465 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:32.937953 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:17:32.937960 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:17:32.938224 (+ 264us) client_negotiation.cc:769] Sending connection context
0928 05:17:32.938440 (+ 216us) client_negotiation.cc:240] Negotiation successful
0928 05:17:32.938881 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":304,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260928 05:17:32.941278 4314 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45007
I20260928 05:17:32.941576 4314 heartbeater.cc:464] Registering TS with master...
I20260928 05:17:32.942284 4314 heartbeater.cc:511] Master 127.0.97.62:45007 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260928 05:17:32.943634 3991 ts_manager.cc:194] Registered new tserver with Master: 5db6e0417ab64f7f86d8e9a68430fa2b (127.0.97.2:43625)
I20260928 05:17:32.944406 3991 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:41195
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572622576160-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572622576160-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260928 05:17:32.964049 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572622576160-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572622576160-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572622576160-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572622576160-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45007
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572622576160-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:17:33.257351 4319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:17:33.257602 4319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:17:33.257730 4319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:17:33.264816 4319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:17:33.264957 4319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:17:33.265137 4319 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 05:17:33.274544 4319 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572622576160-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572622576160-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572622576160-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572622576160-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45007
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572622576160-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:17:33.275718 4319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:17:33.276959 4319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:17:33.288653 4325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:33.288769 4324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:33.288694 4327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:17:33.289475 4319 server_base.cc:1034] running on GCE node
I20260928 05:17:33.290061 4319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:17:33.290874 4319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:17:33.292085 4319 hybrid_clock.cc:648] HybridClock initialized: now 1790572653292045 us; error 33 us; skew 500 ppm
I20260928 05:17:33.292358 4319 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:17:33 dist-test-slave-whsn krb5kdc[3946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572653, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:17:33.295137 4319 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260928 05:17:33.296341 4319 webserver.cc:460] Webserver started at http://127.0.97.3:35359/ using document root <none> and password file <none>
I20260928 05:17:33.296948 4319 fs_manager.cc:359] Metadata directory not provided
I20260928 05:17:33.297053 4319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:17:33.297315 4319 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:17:33.299932 4319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572622576160-388-0/minicluster-data/ts-2/data/instance:
uuid: "0d1cc847a7c04bceada9ec48481f768d"
format_stamp: "Formatted at 2026-09-28 05:17:33 on dist-test-slave-whsn"
server_key: "3ebd65cc0156dbe04e6b2a837654c7f6"
server_key_iv: "a7561aafb5f13b00c77a88fec4dd5258"
server_key_version: "encryptionkey@0"
I20260928 05:17:33.300613 4319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572622576160-388-0/minicluster-data/ts-2/wal/instance:
uuid: "0d1cc847a7c04bceada9ec48481f768d"
format_stamp: "Formatted at 2026-09-28 05:17:33 on dist-test-slave-whsn"
server_key: "3ebd65cc0156dbe04e6b2a837654c7f6"
server_key_iv: "a7561aafb5f13b00c77a88fec4dd5258"
server_key_version: "encryptionkey@0"
I20260928 05:17:33.305192 4319 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 05:17:33.308202 4334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:33.309428 4319 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 05:17:33.309553 4319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572622576160-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572622576160-388-0/minicluster-data/ts-2/wal
uuid: "0d1cc847a7c04bceada9ec48481f768d"
format_stamp: "Formatted at 2026-09-28 05:17:33 on dist-test-slave-whsn"
server_key: "3ebd65cc0156dbe04e6b2a837654c7f6"
server_key_iv: "a7561aafb5f13b00c77a88fec4dd5258"
server_key_version: "encryptionkey@0"
I20260928 05:17:33.309710 4319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572622576160-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572622576160-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572622576160-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:17:33.324535 4319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:17:33.327952 4319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:17:33.328190 4319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:17:33.329022 4319 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:17:33.330178 4319 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:17:33.330243 4319 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:33.330348 4319 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:17:33.330394 4319 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:33.357393 4319 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37827
I20260928 05:17:33.357445 4446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37827 every 8 connection(s)
I20260928 05:17:33.358489 4319 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572622576160-388-0/minicluster-data/ts-2/data/info.pb
I20260928 05:17:33.367129 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 4319
I20260928 05:17:33.367269 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572622576160-388-0/minicluster-data/ts-2/wal/instance
I20260928 05:17:33.367771 388 external_mini_cluster.cc:1428] Setting key 14974fe62b7cf1ca644100a95c7eeddc
Sep 28 05:17:33 dist-test-slave-whsn krb5kdc[3946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572653, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:17:33.376585 4182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:33.360736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48457 (local address 127.0.97.62:45007)
0928 05:17:33.360941 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:33.360947 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:33.362011 (+ 1064us) server_negotiation.cc:407] Connection header received
0928 05:17:33.363621 (+ 1610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:33.363625 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:33.363686 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:33.363768 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:33.365198 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:33.365946 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:33.367413 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:33.368060 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:33.368191 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:33.371510 (+ 3319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:33.371516 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:33.371521 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:33.371551 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:33.373991 (+ 2440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:33.374616 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:33.374621 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:33.374623 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:33.374735 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:33.375103 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:33.375109 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:33.375112 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:33.375359 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:33.375480 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:33.376076 (+ 596us) server_negotiation.cc:299] Negotiation successful
0928 05:17:33.376271 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 05:17:33.377384 4449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:33.361209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45007 (local address 127.0.97.3:48457)
0928 05:17:33.361865 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:33.361906 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:33.363404 (+ 1498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:33.363943 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:33.363951 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:33.364467 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:17:33.365047 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:33.365072 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:33.366103 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:33.366108 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:33.367282 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:33.367297 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:33.368251 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:33.368257 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:33.368554 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:33.369132 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:17:33.369159 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:17:33.371311 (+ 2152us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:17:33.374102 (+ 2791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:33.374110 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:33.374143 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:33.374505 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:33.374827 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:33.374832 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:33.374836 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:33.374999 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:33.375474 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:17:33.375481 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:17:33.375749 (+ 268us) client_negotiation.cc:769] Sending connection context
0928 05:17:33.375955 (+ 206us) client_negotiation.cc:240] Negotiation successful
0928 05:17:33.376378 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":270,"threads_started":1}
I20260928 05:17:33.378569 4447 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45007
I20260928 05:17:33.378794 4447 heartbeater.cc:464] Registering TS with master...
I20260928 05:17:33.379407 4447 heartbeater.cc:511] Master 127.0.97.62:45007 requested a full tablet report, sending...
I20260928 05:17:33.380595 3991 ts_manager.cc:194] Registered new tserver with Master: 0d1cc847a7c04bceada9ec48481f768d (127.0.97.3:37827)
I20260928 05:17:33.381474 3991 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:48457
I20260928 05:17:33.382767 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 05:17:33 dist-test-slave-whsn krb5kdc[3946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572653, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 05:17:33 dist-test-slave-whsn krb5kdc[3946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572653, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260928 05:17:33.423589 4454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:33.410481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50958 (local address 127.0.97.1:44109)
0928 05:17:33.410957 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:33.410965 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:33.410997 (+ 32us) server_negotiation.cc:407] Connection header received
0928 05:17:33.411179 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:33.411185 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:33.411392 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:33.411539 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:33.412795 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:33.413759 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:33.414889 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:33.415392 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:33.415506 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:33.418895 (+ 3389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:33.418904 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:33.418919 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:33.418950 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:33.421525 (+ 2575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:33.422042 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:33.422048 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:33.422050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:33.422150 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:33.422426 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:33.422429 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:33.422431 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:33.422712 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:33.422904 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:33.423097 (+ 193us) server_negotiation.cc:299] Negotiation successful
0928 05:17:33.423258 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":210,"threads_started":1}
Sep 28 05:17:33 dist-test-slave-whsn krb5kdc[3946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572653, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:17:33.438524 4182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:33.427155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33026 (local address 127.0.97.62:45007)
0928 05:17:33.427344 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:33.427349 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:33.427508 (+ 159us) server_negotiation.cc:407] Connection header received
0928 05:17:33.427624 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:33.427628 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:33.427683 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:33.427753 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:33.428721 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:33.429509 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:33.430596 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:33.430975 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:33.431055 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:33.434319 (+ 3264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:33.434324 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:33.434327 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:33.434356 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:33.436570 (+ 2214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:33.437128 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:33.437132 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:33.437134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:33.437241 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:33.437480 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:33.437482 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:33.437484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:33.437747 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:33.437918 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:33.438112 (+ 194us) server_negotiation.cc:299] Negotiation successful
0928 05:17:33.438246 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260928 05:17:33.438985 3991 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33026
I20260928 05:17:33.439821 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 4052
I20260928 05:17:33.450208 4454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:33.440587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50970 (local address 127.0.97.1:44109)
0928 05:17:33.440760 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:33.440765 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:33.440928 (+ 163us) server_negotiation.cc:407] Connection header received
0928 05:17:33.441045 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:33.441048 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:33.441102 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:33.441171 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:33.442139 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:33.442762 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:33.443824 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:33.444197 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:33.444279 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:33.446095 (+ 1816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:33.446099 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:33.446102 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:33.446129 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:33.448351 (+ 2222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:33.448823 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:33.448826 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:33.448828 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:33.448957 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:33.449229 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:33.449232 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:33.449234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:33.449434 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:33.449587 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:33.449795 (+ 208us) server_negotiation.cc:299] Negotiation successful
0928 05:17:33.449933 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 05:17:33.450735 4175 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:17:33.529237 4180 heartbeater.cc:503] Master 127.0.97.62:45007 was elected leader, sending a full tablet report...
I20260928 05:17:33.529803 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 4052
I20260928 05:17:33.542697 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 4186
W20260928 05:17:33.542719 3975 connection.cc:535] server connection from 127.0.97.1:39059 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Sep 28 05:17:33 dist-test-slave-whsn krb5kdc[3946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572653, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260928 05:17:33.556993 4459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:33.543464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46822 (local address 127.0.97.2:43625)
0928 05:17:33.543966 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:33.543972 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:33.543995 (+ 23us) server_negotiation.cc:407] Connection header received
0928 05:17:33.544049 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:33.544056 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:33.544272 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:33.544407 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:33.545645 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:33.546616 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:33.547831 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:33.548346 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:33.548463 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:33.552105 (+ 3642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:33.552114 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:33.552127 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:33.552164 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:33.554511 (+ 2347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:33.555093 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:33.555100 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:33.555102 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:33.555212 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:33.555545 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:33.555551 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:33.555554 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:33.555855 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:33.556137 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:33.556410 (+ 273us) server_negotiation.cc:299] Negotiation successful
0928 05:17:33.556654 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":246,"threads_started":1}
I20260928 05:17:33.558125 4309 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:17:33.636703 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 4186
I20260928 05:17:33.648757 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 4319
Sep 28 05:17:33 dist-test-slave-whsn krb5kdc[3946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572653, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260928 05:17:33.663125 4461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:33.649651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55422 (local address 127.0.97.3:37827)
0928 05:17:33.650207 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:33.650214 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:33.650240 (+ 26us) server_negotiation.cc:407] Connection header received
0928 05:17:33.650297 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:33.650303 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:33.650518 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:33.650656 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:33.651862 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:33.652803 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:33.654056 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:33.654590 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:33.654697 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:33.658507 (+ 3810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:33.658517 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:33.658530 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:33.658560 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:33.661055 (+ 2495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:33.661578 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:33.661584 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:33.661586 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:33.661689 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:33.661938 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:33.661940 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:33.661942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:33.662240 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:33.662452 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:33.662617 (+ 165us) server_negotiation.cc:299] Negotiation successful
0928 05:17:33.662785 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":295,"threads_started":1}
I20260928 05:17:33.664265 4442 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:17:33.743125 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 4319
I20260928 05:17:33.755605 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 3959
I20260928 05:17:33.756938 4021 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:17:33.856110 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 3959
2026-09-28T05:17:33Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3448 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572622576160-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:17:33 dist-test-slave-whsn krb5kdc[4466](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:17:33 dist-test-slave-whsn krb5kdc[4466](info): set up 2 sockets
Sep 28 05:17:33 dist-test-slave-whsn krb5kdc[4466](info): commencing operation
krb5kdc: starting...
W20260928 05:17:34.841106 388 mini_kdc.cc:121] Time spent starting KDC: real 0.949s user 0.002s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572622576160-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572622576160-388-0/krb5kdc/test-user.keytab.
Sep 28 05:17:34 dist-test-slave-whsn krb5kdc[4466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572654, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:17:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:17:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 05:17:35.001660 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572622576160-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572622576160-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572622576160-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40779
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40779
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:17:35.294230 4479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:17:35.294497 4479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:17:35.294627 4479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:17:35.301411 4479 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:17:35.301509 4479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:17:35.301582 4479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:17:35.301651 4479 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:17:35.301715 4479 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:17:35.311152 4479 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572622576160-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572622576160-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40779
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572622576160-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:17:35.312328 4479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:17:35.313568 4479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:17:35.323829 4484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:35.323864 4487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:35.323794 4485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:17:35.324545 4479 server_base.cc:1034] running on GCE node
I20260928 05:17:35.325325 4479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:17:35.326506 4479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:17:35.327696 4479 hybrid_clock.cc:648] HybridClock initialized: now 1790572655327657 us; error 35 us; skew 500 ppm
I20260928 05:17:35.327993 4479 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:17:35 dist-test-slave-whsn krb5kdc[4466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572655, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:17:35.330709 4479 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 05:17:35.331712 4479 fs_manager.cc:359] Metadata directory not provided
I20260928 05:17:35.331806 4479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:17:35.332089 4479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:17:35.334826 4479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572622576160-388-0/minicluster-data/master-0/data/instance:
uuid: "7f02e89feecb417597e5421d43aded8f"
format_stamp: "Formatted at 2026-09-28 05:17:35 on dist-test-slave-whsn"
server_key: "e38b13fd3e217f246d591ea40d3442cc"
server_key_iv: "6f0d7272c7bf6c562d7a055f3fb8e54a"
server_key_version: "encryptionkey@0"
I20260928 05:17:35.335580 4479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572622576160-388-0/minicluster-data/master-0/wal/instance:
uuid: "7f02e89feecb417597e5421d43aded8f"
format_stamp: "Formatted at 2026-09-28 05:17:35 on dist-test-slave-whsn"
server_key: "e38b13fd3e217f246d591ea40d3442cc"
server_key_iv: "6f0d7272c7bf6c562d7a055f3fb8e54a"
server_key_version: "encryptionkey@0"
I20260928 05:17:35.340006 4479 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 05:17:35.343009 4493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:35.344139 4479 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260928 05:17:35.344277 4479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572622576160-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572622576160-388-0/minicluster-data/master-0/wal
uuid: "7f02e89feecb417597e5421d43aded8f"
format_stamp: "Formatted at 2026-09-28 05:17:35 on dist-test-slave-whsn"
server_key: "e38b13fd3e217f246d591ea40d3442cc"
server_key_iv: "6f0d7272c7bf6c562d7a055f3fb8e54a"
server_key_version: "encryptionkey@0"
I20260928 05:17:35.344460 4479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572622576160-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572622576160-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572622576160-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:17:35.358026 4479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:17:35.361310 4479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:17:35.361536 4479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:17:35.377430 4479 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40779
I20260928 05:17:35.377466 4544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40779 every 8 connection(s)
I20260928 05:17:35.378229 4479 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:17:35.381539 4545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:17:35.384186 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 4479
I20260928 05:17:35.384352 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572622576160-388-0/minicluster-data/master-0/wal/instance
I20260928 05:17:35.384984 388 external_mini_cluster.cc:1428] Setting key c9a139d7140b550e4773348e271e68e6
I20260928 05:17:35.392808 4545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f02e89feecb417597e5421d43aded8f: Bootstrap starting.
I20260928 05:17:35.395699 4545 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f02e89feecb417597e5421d43aded8f: Neither blocks nor log segments found. Creating new log.
I20260928 05:17:35.396597 4545 log.cc:824] T 00000000000000000000000000000000 P 7f02e89feecb417597e5421d43aded8f: Log is configured to *not* fsync() on all Append() calls
Sep 28 05:17:35 dist-test-slave-whsn krb5kdc[4466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572654, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:17:35.399415 4545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f02e89feecb417597e5421d43aded8f: No bootstrap required, opened a new log
I20260928 05:17:35.406662 4548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:35.386724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54732 (local address 127.0.97.62:40779)
0928 05:17:35.387461 (+ 737us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:35.387472 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:35.387510 (+ 38us) server_negotiation.cc:407] Connection header received
0928 05:17:35.389076 (+ 1566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:35.389098 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:35.389489 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:35.389805 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:35.391519 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:35.392764 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:35.394059 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:35.394779 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:35.394943 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:35.398914 (+ 3971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:35.398923 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:35.398936 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:35.398973 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:35.402435 (+ 3462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:35.403061 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:35.403069 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:35.403077 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:35.403283 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:35.403619 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:35.403624 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:35.403631 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:35.404315 (+ 684us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:35.404605 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:35.404936 (+ 331us) server_negotiation.cc:299] Negotiation successful
0928 05:17:35.405373 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":348,"threads_started":1}
I20260928 05:17:35.409456 4545 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f02e89feecb417597e5421d43aded8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f02e89feecb417597e5421d43aded8f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40779 } }
I20260928 05:17:35.409772 4545 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7f02e89feecb417597e5421d43aded8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:17:35.409842 4545 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f02e89feecb417597e5421d43aded8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f02e89feecb417597e5421d43aded8f, State: Initialized, Role: FOLLOWER
I20260928 05:17:35.410494 4545 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f02e89feecb417597e5421d43aded8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f02e89feecb417597e5421d43aded8f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40779 } }
I20260928 05:17:35.410789 4550 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f02e89feecb417597e5421d43aded8f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:17:35.411000 4550 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f02e89feecb417597e5421d43aded8f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f02e89feecb417597e5421d43aded8f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40779 } }
I20260928 05:17:35.410760 4545 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7f02e89feecb417597e5421d43aded8f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:17:35.411550 4545 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f02e89feecb417597e5421d43aded8f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:17:35.411684 4545 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f02e89feecb417597e5421d43aded8f [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:17:35.411573 4550 leader_election.cc:302] T 00000000000000000000000000000000 P 7f02e89feecb417597e5421d43aded8f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f02e89feecb417597e5421d43aded8f; no voters:
I20260928 05:17:35.412256 4550 leader_election.cc:288] T 00000000000000000000000000000000 P 7f02e89feecb417597e5421d43aded8f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers
I20260928 05:17:35.413939 4545 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f02e89feecb417597e5421d43aded8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f02e89feecb417597e5421d43aded8f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40779 } }
I20260928 05:17:35.414142 4545 leader_election.cc:302] T 00000000000000000000000000000000 P 7f02e89feecb417597e5421d43aded8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f02e89feecb417597e5421d43aded8f; no voters:
I20260928 05:17:35.414212 4551 raft_consensus.cc:2741] T 00000000000000000000000000000000 P 7f02e89feecb417597e5421d43aded8f [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260928 05:17:35.414336 4545 leader_election.cc:288] T 00000000000000000000000000000000 P 7f02e89feecb417597e5421d43aded8f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:17:35.414605 4550 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f02e89feecb417597e5421d43aded8f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:17:35.415048 4550 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7f02e89feecb417597e5421d43aded8f [term 1 LEADER]: Becoming Leader. State: Replica: 7f02e89feecb417597e5421d43aded8f, State: Running, Role: LEADER
I20260928 05:17:35.415634 4550 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f02e89feecb417597e5421d43aded8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f02e89feecb417597e5421d43aded8f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40779 } }
I20260928 05:17:35.416875 4545 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f02e89feecb417597e5421d43aded8f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:17:35.417685 4551 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f02e89feecb417597e5421d43aded8f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f02e89feecb417597e5421d43aded8f. Latest consensus state: current_term: 1 leader_uuid: "7f02e89feecb417597e5421d43aded8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f02e89feecb417597e5421d43aded8f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40779 } } }
I20260928 05:17:35.417783 4553 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f02e89feecb417597e5421d43aded8f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7f02e89feecb417597e5421d43aded8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f02e89feecb417597e5421d43aded8f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40779 } } }
I20260928 05:17:35.418077 4551 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f02e89feecb417597e5421d43aded8f [sys.catalog]: This master's current role is: LEADER
I20260928 05:17:35.418087 4553 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f02e89feecb417597e5421d43aded8f [sys.catalog]: This master's current role is: LEADER
I20260928 05:17:35.418869 4556 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:17:35.424130 4556 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:17:35.435050 4556 catalog_manager.cc:1347] Generated new cluster ID: 262554c984f847b29f1493390be4f7e6
I20260928 05:17:35.435140 4556 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:17:35.448913 4556 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:17:35.450074 4556 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:17:35.456986 4556 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7f02e89feecb417597e5421d43aded8f: Generated new TSK 0
I20260928 05:17:35.457690 4556 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572622576160-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572622576160-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260928 05:17:35.502432 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572622576160-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572622576160-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572622576160-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572622576160-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40779
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572622576160-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:17:35.823501 4572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:17:35.823745 4572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:17:35.823879 4572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:17:35.830693 4572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:17:35.830778 4572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:17:35.830921 4572 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 05:17:35.839826 4572 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572622576160-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572622576160-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572622576160-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572622576160-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40779
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572622576160-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:17:35.840976 4572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:17:35.842139 4572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:17:35.853626 4577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:35.853662 4578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:35.853842 4580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:17:35.854427 4572 server_base.cc:1034] running on GCE node
I20260928 05:17:35.855003 4572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:17:35.855801 4572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:17:35.856926 4572 hybrid_clock.cc:648] HybridClock initialized: now 1790572655856872 us; error 46 us; skew 500 ppm
I20260928 05:17:35.857203 4572 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:17:35 dist-test-slave-whsn krb5kdc[4466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572655, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:17:35.859867 4572 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260928 05:17:35.860904 4572 fs_manager.cc:359] Metadata directory not provided
I20260928 05:17:35.861006 4572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:17:35.861287 4572 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:17:35.863871 4572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572622576160-388-0/minicluster-data/ts-0/data/instance:
uuid: "f2d94c8bcba3402d906415ca372c0523"
format_stamp: "Formatted at 2026-09-28 05:17:35 on dist-test-slave-whsn"
server_key: "0dab196e9a169a8e968cb9694bf841ef"
server_key_iv: "a94aa1a69fb2b70fb0034a1d622d223e"
server_key_version: "encryptionkey@0"
I20260928 05:17:35.864678 4572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572622576160-388-0/minicluster-data/ts-0/wal/instance:
uuid: "f2d94c8bcba3402d906415ca372c0523"
format_stamp: "Formatted at 2026-09-28 05:17:35 on dist-test-slave-whsn"
server_key: "0dab196e9a169a8e968cb9694bf841ef"
server_key_iv: "a94aa1a69fb2b70fb0034a1d622d223e"
server_key_version: "encryptionkey@0"
I20260928 05:17:35.869062 4572 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260928 05:17:35.872051 4586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:35.873229 4572 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 05:17:35.873363 4572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572622576160-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572622576160-388-0/minicluster-data/ts-0/wal
uuid: "f2d94c8bcba3402d906415ca372c0523"
format_stamp: "Formatted at 2026-09-28 05:17:35 on dist-test-slave-whsn"
server_key: "0dab196e9a169a8e968cb9694bf841ef"
server_key_iv: "a94aa1a69fb2b70fb0034a1d622d223e"
server_key_version: "encryptionkey@0"
I20260928 05:17:35.873546 4572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572622576160-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572622576160-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572622576160-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:17:35.888127 4572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:17:35.891528 4572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:17:35.891745 4572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:17:35.892568 4572 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:17:35.893770 4572 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:17:35.893869 4572 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:35.893970 4572 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:17:35.894040 4572 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:35.922300 4572 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41471
I20260928 05:17:35.922351 4698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41471 every 8 connection(s)
I20260928 05:17:35.923065 4572 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572622576160-388-0/minicluster-data/ts-0/data/info.pb
I20260928 05:17:35.927879 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 4572
I20260928 05:17:35.928072 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572622576160-388-0/minicluster-data/ts-0/wal/instance
I20260928 05:17:35.928833 388 external_mini_cluster.cc:1428] Setting key 27813344b03cb0a4bca6934361d26bc5
Sep 28 05:17:35 dist-test-slave-whsn krb5kdc[4466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572655, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:17:35.942502 4701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:35.925153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44229 (local address 127.0.97.62:40779)
0928 05:17:35.925530 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:35.925534 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:35.926357 (+ 823us) server_negotiation.cc:407] Connection header received
0928 05:17:35.928265 (+ 1908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:35.928269 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:35.928333 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:35.928422 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:35.930164 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:35.930915 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:35.933049 (+ 2134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:35.933411 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:35.933507 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:35.936950 (+ 3443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:35.936956 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:35.936961 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:35.936996 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:35.939798 (+ 2802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:35.940412 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:35.940417 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:35.940420 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:35.940552 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:35.940947 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:35.940951 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:35.940953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:35.941219 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:35.941345 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:35.941940 (+ 595us) server_negotiation.cc:299] Negotiation successful
0928 05:17:35.942151 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":190,"threads_started":1}
I20260928 05:17:35.943426 4702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:35.925578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40779 (local address 127.0.97.1:44229)
0928 05:17:35.926214 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:35.926256 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:35.927982 (+ 1726us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:35.928675 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:35.928686 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:35.929397 (+ 711us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:17:35.930015 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:35.930039 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:35.931065 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:35.931070 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:35.932917 (+ 1847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:35.932931 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:35.933609 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:35.933616 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:35.933853 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:35.934491 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:17:35.934523 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:17:35.936718 (+ 2195us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:17:35.939920 (+ 3202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:35.939926 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:35.939936 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:35.940315 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:35.940663 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:35.940667 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:35.940671 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:35.940828 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:35.941333 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:17:35.941340 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:17:35.941621 (+ 281us) client_negotiation.cc:769] Sending connection context
0928 05:17:35.941837 (+ 216us) client_negotiation.cc:240] Negotiation successful
0928 05:17:35.942299 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":246,"threads_started":1}
I20260928 05:17:35.944988 4699 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40779
I20260928 05:17:35.945236 4699 heartbeater.cc:464] Registering TS with master...
I20260928 05:17:35.945806 4699 heartbeater.cc:511] Master 127.0.97.62:40779 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260928 05:17:35.947723 4510 ts_manager.cc:194] Registered new tserver with Master: f2d94c8bcba3402d906415ca372c0523 (127.0.97.1:41471)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260928 05:17:35.949223 4510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:44229
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572622576160-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572622576160-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260928 05:17:35.967581 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572622576160-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572622576160-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572622576160-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572622576160-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40779
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572622576160-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:17:36.270860 4705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:17:36.271149 4705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:17:36.271289 4705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:17:36.278340 4705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:17:36.278447 4705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:17:36.278599 4705 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 05:17:36.287953 4705 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572622576160-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572622576160-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572622576160-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572622576160-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40779
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572622576160-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:17:36.289168 4705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:17:36.290438 4705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:17:36.302279 4711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:36.302353 4710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:36.302363 4713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:17:36.302896 4705 server_base.cc:1034] running on GCE node
I20260928 05:17:36.303417 4705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:17:36.305001 4705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:17:36.306200 4705 hybrid_clock.cc:648] HybridClock initialized: now 1790572656306146 us; error 44 us; skew 500 ppm
I20260928 05:17:36.306500 4705 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:17:36 dist-test-slave-whsn krb5kdc[4466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572656, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:17:36.309424 4705 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260928 05:17:36.310407 4705 fs_manager.cc:359] Metadata directory not provided
I20260928 05:17:36.310515 4705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:17:36.310789 4705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:17:36.313485 4705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572622576160-388-0/minicluster-data/ts-1/data/instance:
uuid: "1dbd21d244bc4eacb6d7ac79c32fbd51"
format_stamp: "Formatted at 2026-09-28 05:17:36 on dist-test-slave-whsn"
server_key: "f5094278a29bd2ac06096db5095ad8bf"
server_key_iv: "e385d24becdf274869e9001db3fd1cca"
server_key_version: "encryptionkey@0"
I20260928 05:17:36.314263 4705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572622576160-388-0/minicluster-data/ts-1/wal/instance:
uuid: "1dbd21d244bc4eacb6d7ac79c32fbd51"
format_stamp: "Formatted at 2026-09-28 05:17:36 on dist-test-slave-whsn"
server_key: "f5094278a29bd2ac06096db5095ad8bf"
server_key_iv: "e385d24becdf274869e9001db3fd1cca"
server_key_version: "encryptionkey@0"
I20260928 05:17:36.319007 4705 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260928 05:17:36.322288 4719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:36.323542 4705 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 05:17:36.323683 4705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572622576160-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572622576160-388-0/minicluster-data/ts-1/wal
uuid: "1dbd21d244bc4eacb6d7ac79c32fbd51"
format_stamp: "Formatted at 2026-09-28 05:17:36 on dist-test-slave-whsn"
server_key: "f5094278a29bd2ac06096db5095ad8bf"
server_key_iv: "e385d24becdf274869e9001db3fd1cca"
server_key_version: "encryptionkey@0"
I20260928 05:17:36.323864 4705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572622576160-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572622576160-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572622576160-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:17:36.359022 4705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:17:36.362695 4705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:17:36.362918 4705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:17:36.363816 4705 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:17:36.365121 4705 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:17:36.365213 4705 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:36.365312 4705 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:17:36.365377 4705 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:36.394946 4705 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40579
I20260928 05:17:36.395010 4831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40579 every 8 connection(s)
I20260928 05:17:36.395740 4705 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572622576160-388-0/minicluster-data/ts-1/data/info.pb
I20260928 05:17:36.402046 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 4705
I20260928 05:17:36.402191 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572622576160-388-0/minicluster-data/ts-1/wal/instance
I20260928 05:17:36.402592 388 external_mini_cluster.cc:1428] Setting key df23685288b1f8862c23479f2370f295
Sep 28 05:17:36 dist-test-slave-whsn krb5kdc[4466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572656, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:17:36.414630 4701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:36.397959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53175 (local address 127.0.97.62:40779)
0928 05:17:36.398140 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:36.398146 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:36.399260 (+ 1114us) server_negotiation.cc:407] Connection header received
0928 05:17:36.401091 (+ 1831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:36.401095 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:36.401159 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:36.401240 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:36.402650 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:36.403482 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:36.405104 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:36.405507 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:36.405614 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:36.408990 (+ 3376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:36.408997 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:36.409002 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:36.409040 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:36.411928 (+ 2888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:36.412556 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:36.412561 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:36.412565 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:36.412714 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:36.413106 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:36.413110 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:36.413113 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:36.413425 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:36.413554 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:36.414120 (+ 566us) server_negotiation.cc:299] Negotiation successful
0928 05:17:36.414320 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 05:17:36.415426 4834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:36.398463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40779 (local address 127.0.97.2:53175)
0928 05:17:36.399119 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:36.399159 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:36.400808 (+ 1649us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:36.401407 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:36.401415 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:36.401911 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:17:36.402489 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:36.402506 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:36.403634 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:36.403638 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:36.404974 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:36.404989 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:36.405704 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:36.405711 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:36.405917 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:36.406510 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:17:36.406532 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:17:36.408741 (+ 2209us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:17:36.412057 (+ 3316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:36.412064 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:36.412077 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:36.412457 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:36.412807 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:36.412810 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:36.412812 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:36.413016 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:36.413521 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:17:36.413526 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:17:36.413798 (+ 272us) client_negotiation.cc:769] Sending connection context
0928 05:17:36.414027 (+ 229us) client_negotiation.cc:240] Negotiation successful
0928 05:17:36.414373 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":261,"threads_started":1}
I20260928 05:17:36.416723 4832 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40779
I20260928 05:17:36.417025 4832 heartbeater.cc:464] Registering TS with master...
I20260928 05:17:36.417631 4832 heartbeater.cc:511] Master 127.0.97.62:40779 requested a full tablet report, sending...
I20260928 05:17:36.418877 4510 ts_manager.cc:194] Registered new tserver with Master: 1dbd21d244bc4eacb6d7ac79c32fbd51 (127.0.97.2:40579)
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260928 05:17:36.419636 4510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:53175
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572622576160-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572622576160-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260928 05:17:36.439915 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572622576160-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572622576160-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572622576160-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572622576160-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40779
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572622576160-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:17:36.731433 4837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:17:36.731665 4837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:17:36.731786 4837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:17:36.738551 4837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:17:36.738636 4837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:17:36.738785 4837 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 05:17:36.747730 4837 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572622576160-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572622576160-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572622576160-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572622576160-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40779
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572622576160-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:17:36.748881 4837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:17:36.750057 4837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:17:36.761263 4843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:36.761339 4842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:36.761432 4845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:17:36.762233 4837 server_base.cc:1034] running on GCE node
I20260928 05:17:36.762732 4837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:17:36.763517 4837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:17:36.764704 4837 hybrid_clock.cc:648] HybridClock initialized: now 1790572656764653 us; error 42 us; skew 500 ppm
I20260928 05:17:36.765029 4837 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:17:36 dist-test-slave-whsn krb5kdc[4466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572656, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:17:36.767853 4837 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260928 05:17:36.768826 4837 fs_manager.cc:359] Metadata directory not provided
I20260928 05:17:36.768944 4837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:17:36.769238 4837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:17:36.771812 4837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572622576160-388-0/minicluster-data/ts-2/data/instance:
uuid: "5f03f228a75347008bdf55802a49afca"
format_stamp: "Formatted at 2026-09-28 05:17:36 on dist-test-slave-whsn"
server_key: "0a2d0891837612ec44a68e8abcda4edf"
server_key_iv: "d20d0d03c22d0ec094a3aae0ba868abe"
server_key_version: "encryptionkey@0"
I20260928 05:17:36.772537 4837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572622576160-388-0/minicluster-data/ts-2/wal/instance:
uuid: "5f03f228a75347008bdf55802a49afca"
format_stamp: "Formatted at 2026-09-28 05:17:36 on dist-test-slave-whsn"
server_key: "0a2d0891837612ec44a68e8abcda4edf"
server_key_iv: "d20d0d03c22d0ec094a3aae0ba868abe"
server_key_version: "encryptionkey@0"
I20260928 05:17:36.777135 4837 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260928 05:17:36.780196 4851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:36.781428 4837 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 05:17:36.781570 4837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572622576160-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572622576160-388-0/minicluster-data/ts-2/wal
uuid: "5f03f228a75347008bdf55802a49afca"
format_stamp: "Formatted at 2026-09-28 05:17:36 on dist-test-slave-whsn"
server_key: "0a2d0891837612ec44a68e8abcda4edf"
server_key_iv: "d20d0d03c22d0ec094a3aae0ba868abe"
server_key_version: "encryptionkey@0"
I20260928 05:17:36.781750 4837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572622576160-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572622576160-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572622576160-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:17:36.802240 4837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:17:36.806036 4837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:17:36.806263 4837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:17:36.807118 4837 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:17:36.808305 4837 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:17:36.808396 4837 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:36.808495 4837 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:17:36.808564 4837 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:36.836018 4837 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40591
I20260928 05:17:36.836058 4963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40591 every 8 connection(s)
I20260928 05:17:36.836758 4837 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572622576160-388-0/minicluster-data/ts-2/data/info.pb
I20260928 05:17:36.843108 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 4837
I20260928 05:17:36.843292 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572622576160-388-0/minicluster-data/ts-2/wal/instance
I20260928 05:17:36.843770 388 external_mini_cluster.cc:1428] Setting key 200722bba95c38c66e8ca4a096f064f5
Sep 28 05:17:36 dist-test-slave-whsn krb5kdc[4466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572656, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:17:36.854059 4701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:36.838911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42583 (local address 127.0.97.62:40779)
0928 05:17:36.839071 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:36.839075 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:36.840130 (+ 1055us) server_negotiation.cc:407] Connection header received
0928 05:17:36.841809 (+ 1679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:36.841814 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:36.841879 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:36.841958 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:36.843517 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:36.844298 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:36.845842 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:36.846258 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:36.846361 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:36.849473 (+ 3112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:36.849477 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:36.849481 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:36.849510 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:36.851664 (+ 2154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:36.852258 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:36.852262 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:36.852264 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:36.852364 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:36.852671 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:36.852674 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:36.852676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:36.852987 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:36.853106 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:36.853667 (+ 561us) server_negotiation.cc:299] Negotiation successful
0928 05:17:36.853813 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 05:17:36.854866 4966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:36.839316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40779 (local address 127.0.97.3:42583)
0928 05:17:36.839959 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:36.840013 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:36.841561 (+ 1548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:36.842152 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:36.842160 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:36.842618 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:17:36.843364 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:36.843389 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:36.844448 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:36.844452 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:36.845710 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:36.845721 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:36.846474 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:36.846482 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:36.846688 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:36.847262 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:17:36.847282 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:17:36.849296 (+ 2014us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:17:36.851774 (+ 2478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:36.851780 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:36.851791 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:36.852171 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:36.852435 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:36.852437 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:36.852439 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:36.852594 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:36.853087 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:17:36.853092 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:17:36.853358 (+ 266us) client_negotiation.cc:769] Sending connection context
0928 05:17:36.853569 (+ 211us) client_negotiation.cc:240] Negotiation successful
0928 05:17:36.853899 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":259,"threads_started":1}
I20260928 05:17:36.855994 4964 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40779
I20260928 05:17:36.856236 4964 heartbeater.cc:464] Registering TS with master...
I20260928 05:17:36.856786 4964 heartbeater.cc:511] Master 127.0.97.62:40779 requested a full tablet report, sending...
I20260928 05:17:36.858111 4509 ts_manager.cc:194] Registered new tserver with Master: 5f03f228a75347008bdf55802a49afca (127.0.97.3:40591)
I20260928 05:17:36.858875 4509 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:42583
I20260928 05:17:36.869894 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 05:17:36.884654 4701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:36.873930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54740 (local address 127.0.97.62:40779)
0928 05:17:36.874068 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:36.874072 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:36.874556 (+ 484us) server_negotiation.cc:407] Connection header received
0928 05:17:36.874674 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:36.874678 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:36.874735 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:36.874806 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:36.876011 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:36.876747 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:36.878014 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:36.878403 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:36.878491 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:36.880403 (+ 1912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:36.880408 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:36.880411 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:36.880438 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:36.882691 (+ 2253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:36.883207 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:36.883211 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:36.883213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:36.883315 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:36.883564 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:36.883567 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:36.883569 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:36.883823 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:36.884010 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:36.884195 (+ 185us) server_negotiation.cc:299] Negotiation successful
0928 05:17:36.884363 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 05:17:36.888101 4509 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54740:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 05:17:36.894515 4509 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 05:17:36.929533 4978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:36.905484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40591 (local address 127.0.0.1:35442)
0928 05:17:36.918944 (+ 13460us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:36.918987 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:36.919171 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:36.920944 (+ 1773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:36.920953 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:36.920989 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:17:36.921419 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:36.921432 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:36.923654 (+ 2222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:36.923660 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:36.926093 (+ 2433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:36.926110 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:36.928533 (+ 2423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:36.928540 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:36.928767 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:36.928818 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 05:17:36.928963 (+ 145us) client_negotiation.cc:240] Negotiation successful
0928 05:17:36.929149 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13183,"thread_start_us":15326,"threads_started":1}
I20260928 05:17:36.931138 4977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:36.905635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35442 (local address 127.0.97.3:40591)
0928 05:17:36.919979 (+ 14344us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:36.919986 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:36.920020 (+ 34us) server_negotiation.cc:407] Connection header received
0928 05:17:36.920121 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:36.920130 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:36.920405 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:36.920590 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:17:36.921639 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:36.923486 (+ 1847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:36.926314 (+ 2828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:36.928044 (+ 1730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:36.928168 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:36.928398 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:36.930609 (+ 2211us) server_negotiation.cc:299] Negotiation successful
0928 05:17:36.930788 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14184,"thread_start_us":14034,"threads_started":1}
I20260928 05:17:36.935078 4899 tablet_service.cc:1467] Processing CreateTablet for tablet 416c2ab10d004761ab01d2b6854b4a42 (DEFAULT_TABLE table=test-table [id=38e6ae17052040f381227d9b0b1c5372]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:17:36.936363 4979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:36.906180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60732 (local address 127.0.97.2:40579)
0928 05:17:36.918364 (+ 12184us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:36.918375 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:36.920786 (+ 2411us) server_negotiation.cc:407] Connection header received
0928 05:17:36.928961 (+ 8175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:36.928970 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:36.929230 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:36.929391 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:17:36.930713 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:36.932200 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:36.934093 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:36.935218 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:36.935291 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:36.935420 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:36.935940 (+ 520us) server_negotiation.cc:299] Negotiation successful
0928 05:17:36.936098 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11994,"thread_start_us":13128,"threads_started":1}
I20260928 05:17:36.936378 4976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:36.905780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40579 (local address 127.0.0.1:60732)
0928 05:17:36.920665 (+ 14885us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:36.920685 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:36.921110 (+ 425us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:36.930075 (+ 8965us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:36.930080 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:36.930110 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:17:36.930515 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:36.930526 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:36.932320 (+ 1794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:36.932325 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:36.933798 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:36.933809 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:36.935573 (+ 1764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:36.935577 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:36.935779 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:36.935811 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 05:17:36.935887 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 05:17:36.936008 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14742,"mutex_wait_us":28,"thread_start_us":12751,"threads_started":1}
I20260928 05:17:36.936477 4899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 416c2ab10d004761ab01d2b6854b4a42. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:17:36.938560 4978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:36.919961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41471 (local address 127.0.0.1:54898)
0928 05:17:36.930058 (+ 10097us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:36.930076 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:36.930177 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:36.932411 (+ 2234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:36.932419 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:36.932443 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:17:36.932647 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:36.932657 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:36.934397 (+ 1740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:36.934401 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:36.936169 (+ 1768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:36.936182 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:36.937817 (+ 1635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:36.937823 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:36.937991 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:36.938022 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 05:17:36.938116 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 05:17:36.938243 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10002,"mutex_wait_us":24,"thread_start_us":12042,"threads_started":1}
I20260928 05:17:36.938576 4980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:36.919649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54898 (local address 127.0.97.1:41471)
0928 05:17:36.931909 (+ 12260us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:36.931917 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:36.931951 (+ 34us) server_negotiation.cc:407] Connection header received
0928 05:17:36.932025 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:36.932031 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:36.932234 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:36.932405 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:17:36.932789 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:36.934231 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:36.936346 (+ 2115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:36.937672 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:36.937794 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:36.938006 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:36.938163 (+ 157us) server_negotiation.cc:299] Negotiation successful
0928 05:17:36.938306 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12114,"thread_start_us":11985,"threads_started":1}
I20260928 05:17:36.939903 4767 tablet_service.cc:1467] Processing CreateTablet for tablet 416c2ab10d004761ab01d2b6854b4a42 (DEFAULT_TABLE table=test-table [id=38e6ae17052040f381227d9b0b1c5372]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:17:36.940955 4767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 416c2ab10d004761ab01d2b6854b4a42. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:17:36.941884 4634 tablet_service.cc:1467] Processing CreateTablet for tablet 416c2ab10d004761ab01d2b6854b4a42 (DEFAULT_TABLE table=test-table [id=38e6ae17052040f381227d9b0b1c5372]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:17:36.943120 4634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 416c2ab10d004761ab01d2b6854b4a42. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:17:36.950096 4982 tablet_bootstrap.cc:492] T 416c2ab10d004761ab01d2b6854b4a42 P 5f03f228a75347008bdf55802a49afca: Bootstrap starting.
I20260928 05:17:36.951853 4983 tablet_bootstrap.cc:492] T 416c2ab10d004761ab01d2b6854b4a42 P 1dbd21d244bc4eacb6d7ac79c32fbd51: Bootstrap starting.
I20260928 05:17:36.952092 4699 heartbeater.cc:503] Master 127.0.97.62:40779 was elected leader, sending a full tablet report...
I20260928 05:17:36.954331 4982 tablet_bootstrap.cc:654] T 416c2ab10d004761ab01d2b6854b4a42 P 5f03f228a75347008bdf55802a49afca: Neither blocks nor log segments found. Creating new log.
I20260928 05:17:36.955161 4983 tablet_bootstrap.cc:654] T 416c2ab10d004761ab01d2b6854b4a42 P 1dbd21d244bc4eacb6d7ac79c32fbd51: Neither blocks nor log segments found. Creating new log.
I20260928 05:17:36.955440 4982 log.cc:824] T 416c2ab10d004761ab01d2b6854b4a42 P 5f03f228a75347008bdf55802a49afca: Log is configured to *not* fsync() on all Append() calls
I20260928 05:17:36.956317 4983 log.cc:824] T 416c2ab10d004761ab01d2b6854b4a42 P 1dbd21d244bc4eacb6d7ac79c32fbd51: Log is configured to *not* fsync() on all Append() calls
I20260928 05:17:36.957121 4984 tablet_bootstrap.cc:492] T 416c2ab10d004761ab01d2b6854b4a42 P f2d94c8bcba3402d906415ca372c0523: Bootstrap starting.
I20260928 05:17:36.960423 4984 tablet_bootstrap.cc:654] T 416c2ab10d004761ab01d2b6854b4a42 P f2d94c8bcba3402d906415ca372c0523: Neither blocks nor log segments found. Creating new log.
I20260928 05:17:36.961284 4984 log.cc:824] T 416c2ab10d004761ab01d2b6854b4a42 P f2d94c8bcba3402d906415ca372c0523: Log is configured to *not* fsync() on all Append() calls
I20260928 05:17:36.961912 4983 tablet_bootstrap.cc:492] T 416c2ab10d004761ab01d2b6854b4a42 P 1dbd21d244bc4eacb6d7ac79c32fbd51: No bootstrap required, opened a new log
I20260928 05:17:36.962169 4983 ts_tablet_manager.cc:1397] T 416c2ab10d004761ab01d2b6854b4a42 P 1dbd21d244bc4eacb6d7ac79c32fbd51: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260928 05:17:36.963318 4982 tablet_bootstrap.cc:492] T 416c2ab10d004761ab01d2b6854b4a42 P 5f03f228a75347008bdf55802a49afca: No bootstrap required, opened a new log
I20260928 05:17:36.963580 4982 ts_tablet_manager.cc:1397] T 416c2ab10d004761ab01d2b6854b4a42 P 5f03f228a75347008bdf55802a49afca: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.000s
I20260928 05:17:36.963824 4984 tablet_bootstrap.cc:492] T 416c2ab10d004761ab01d2b6854b4a42 P f2d94c8bcba3402d906415ca372c0523: No bootstrap required, opened a new log
I20260928 05:17:36.964067 4984 ts_tablet_manager.cc:1397] T 416c2ab10d004761ab01d2b6854b4a42 P f2d94c8bcba3402d906415ca372c0523: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20260928 05:17:36.969053 4983 raft_consensus.cc:348] T 416c2ab10d004761ab01d2b6854b4a42 P 1dbd21d244bc4eacb6d7ac79c32fbd51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f03f228a75347008bdf55802a49afca" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40591 } } peers { permanent_uuid: "1dbd21d244bc4eacb6d7ac79c32fbd51" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40579 } } peers { permanent_uuid: "f2d94c8bcba3402d906415ca372c0523" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41471 } }
I20260928 05:17:36.969369 4983 raft_consensus.cc:374] T 416c2ab10d004761ab01d2b6854b4a42 P 1dbd21d244bc4eacb6d7ac79c32fbd51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:17:36.969493 4983 raft_consensus.cc:729] T 416c2ab10d004761ab01d2b6854b4a42 P 1dbd21d244bc4eacb6d7ac79c32fbd51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dbd21d244bc4eacb6d7ac79c32fbd51, State: Initialized, Role: FOLLOWER
I20260928 05:17:36.970005 4983 consensus_queue.cc:260] T 416c2ab10d004761ab01d2b6854b4a42 P 1dbd21d244bc4eacb6d7ac79c32fbd51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f03f228a75347008bdf55802a49afca" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40591 } } peers { permanent_uuid: "1dbd21d244bc4eacb6d7ac79c32fbd51" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40579 } } peers { permanent_uuid: "f2d94c8bcba3402d906415ca372c0523" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41471 } }
I20260928 05:17:36.970984 4832 heartbeater.cc:503] Master 127.0.97.62:40779 was elected leader, sending a full tablet report...
I20260928 05:17:36.971508 4983 ts_tablet_manager.cc:1428] T 416c2ab10d004761ab01d2b6854b4a42 P 1dbd21d244bc4eacb6d7ac79c32fbd51: Time spent starting tablet: real 0.009s user 0.009s sys 0.000s
I20260928 05:17:36.973138 4982 raft_consensus.cc:348] T 416c2ab10d004761ab01d2b6854b4a42 P 5f03f228a75347008bdf55802a49afca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f03f228a75347008bdf55802a49afca" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40591 } } peers { permanent_uuid: "1dbd21d244bc4eacb6d7ac79c32fbd51" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40579 } } peers { permanent_uuid: "f2d94c8bcba3402d906415ca372c0523" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41471 } }
I20260928 05:17:36.973536 4982 raft_consensus.cc:374] T 416c2ab10d004761ab01d2b6854b4a42 P 5f03f228a75347008bdf55802a49afca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:17:36.973629 4982 raft_consensus.cc:729] T 416c2ab10d004761ab01d2b6854b4a42 P 5f03f228a75347008bdf55802a49afca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f03f228a75347008bdf55802a49afca, State: Initialized, Role: FOLLOWER
I20260928 05:17:36.974172 4982 consensus_queue.cc:260] T 416c2ab10d004761ab01d2b6854b4a42 P 5f03f228a75347008bdf55802a49afca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f03f228a75347008bdf55802a49afca" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40591 } } peers { permanent_uuid: "1dbd21d244bc4eacb6d7ac79c32fbd51" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40579 } } peers { permanent_uuid: "f2d94c8bcba3402d906415ca372c0523" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41471 } }
I20260928 05:17:36.974550 4984 raft_consensus.cc:348] T 416c2ab10d004761ab01d2b6854b4a42 P f2d94c8bcba3402d906415ca372c0523 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f03f228a75347008bdf55802a49afca" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40591 } } peers { permanent_uuid: "1dbd21d244bc4eacb6d7ac79c32fbd51" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40579 } } peers { permanent_uuid: "f2d94c8bcba3402d906415ca372c0523" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41471 } }
I20260928 05:17:36.974962 4984 raft_consensus.cc:374] T 416c2ab10d004761ab01d2b6854b4a42 P f2d94c8bcba3402d906415ca372c0523 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:17:36.975096 4984 raft_consensus.cc:729] T 416c2ab10d004761ab01d2b6854b4a42 P f2d94c8bcba3402d906415ca372c0523 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2d94c8bcba3402d906415ca372c0523, State: Initialized, Role: FOLLOWER
I20260928 05:17:36.975205 4964 heartbeater.cc:503] Master 127.0.97.62:40779 was elected leader, sending a full tablet report...
I20260928 05:17:36.975835 4982 ts_tablet_manager.cc:1428] T 416c2ab10d004761ab01d2b6854b4a42 P 5f03f228a75347008bdf55802a49afca: Time spent starting tablet: real 0.012s user 0.008s sys 0.005s
I20260928 05:17:36.975769 4984 consensus_queue.cc:260] T 416c2ab10d004761ab01d2b6854b4a42 P f2d94c8bcba3402d906415ca372c0523 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f03f228a75347008bdf55802a49afca" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40591 } } peers { permanent_uuid: "1dbd21d244bc4eacb6d7ac79c32fbd51" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40579 } } peers { permanent_uuid: "f2d94c8bcba3402d906415ca372c0523" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41471 } }
I20260928 05:17:36.977928 4984 ts_tablet_manager.cc:1428] T 416c2ab10d004761ab01d2b6854b4a42 P f2d94c8bcba3402d906415ca372c0523: Time spent starting tablet: real 0.014s user 0.012s sys 0.001s
W20260928 05:17:37.088107 4965 tablet.cc:2369] T 416c2ab10d004761ab01d2b6854b4a42 P 5f03f228a75347008bdf55802a49afca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 05:17:37.093185 4989 raft_consensus.cc:482] T 416c2ab10d004761ab01d2b6854b4a42 P 5f03f228a75347008bdf55802a49afca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:17:37.093431 4989 raft_consensus.cc:504] T 416c2ab10d004761ab01d2b6854b4a42 P 5f03f228a75347008bdf55802a49afca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f03f228a75347008bdf55802a49afca" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40591 } } peers { permanent_uuid: "1dbd21d244bc4eacb6d7ac79c32fbd51" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40579 } } peers { permanent_uuid: "f2d94c8bcba3402d906415ca372c0523" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41471 } }
I20260928 05:17:37.094928 4989 leader_election.cc:288] T 416c2ab10d004761ab01d2b6854b4a42 P 5f03f228a75347008bdf55802a49afca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1dbd21d244bc4eacb6d7ac79c32fbd51 (127.0.97.2:40579), f2d94c8bcba3402d906415ca372c0523 (127.0.97.1:41471)
I20260928 05:17:37.100927 4979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:37.095361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39227 (local address 127.0.97.2:40579)
0928 05:17:37.095578 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:37.095584 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:37.095604 (+ 20us) server_negotiation.cc:407] Connection header received
0928 05:17:37.095674 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:37.095679 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:37.095750 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:37.095845 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:17:37.096413 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:37.097373 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:37.099070 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:37.100144 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:37.100241 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:37.100309 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:37.100634 (+ 325us) server_negotiation.cc:299] Negotiation successful
0928 05:17:37.100717 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 05:17:37.101039 4966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:37.095169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40579 (local address 127.0.97.3:39227)
0928 05:17:37.095375 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:37.095394 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:37.095504 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:37.096069 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:37.096073 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:37.096104 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:17:37.096287 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:37.096297 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:37.097503 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:37.097507 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:37.098894 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:37.098905 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:37.100286 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:37.100292 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:37.100470 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:37.100514 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 05:17:37.100615 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 05:17:37.100744 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":1}
I20260928 05:17:37.101855 4991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:37.095606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41471 (local address 127.0.97.3:39193)
0928 05:17:37.096138 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:37.096157 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:37.096264 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:37.096549 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:37.096552 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:37.096571 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:17:37.096811 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:37.096817 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:37.098117 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:37.098121 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:37.099734 (+ 1613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:37.099748 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:37.101278 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:37.101284 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:37.101436 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:37.101464 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 05:17:37.101555 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 05:17:37.101654 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":281,"threads_started":1}
I20260928 05:17:37.101927 4980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:37.095773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39193 (local address 127.0.97.1:41471)
0928 05:17:37.095955 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:37.095961 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:37.096235 (+ 274us) server_negotiation.cc:407] Connection header received
0928 05:17:37.096371 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:37.096375 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:37.096448 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:37.096539 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:17:37.097045 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:37.097943 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:37.099892 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:37.100925 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:37.101057 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:37.101147 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:37.101589 (+ 442us) server_negotiation.cc:299] Negotiation successful
0928 05:17:37.101714 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 05:17:37.102032 4787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "416c2ab10d004761ab01d2b6854b4a42" candidate_uuid: "5f03f228a75347008bdf55802a49afca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dbd21d244bc4eacb6d7ac79c32fbd51" is_pre_election: true
I20260928 05:17:37.102557 4787 raft_consensus.cc:2454] T 416c2ab10d004761ab01d2b6854b4a42 P 1dbd21d244bc4eacb6d7ac79c32fbd51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f03f228a75347008bdf55802a49afca in term 0.
I20260928 05:17:37.102910 4654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "416c2ab10d004761ab01d2b6854b4a42" candidate_uuid: "5f03f228a75347008bdf55802a49afca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2d94c8bcba3402d906415ca372c0523" is_pre_election: true
I20260928 05:17:37.103159 4853 leader_election.cc:302] T 416c2ab10d004761ab01d2b6854b4a42 P 5f03f228a75347008bdf55802a49afca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dbd21d244bc4eacb6d7ac79c32fbd51, 5f03f228a75347008bdf55802a49afca; no voters:
I20260928 05:17:37.103390 4654 raft_consensus.cc:2454] T 416c2ab10d004761ab01d2b6854b4a42 P f2d94c8bcba3402d906415ca372c0523 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f03f228a75347008bdf55802a49afca in term 0.
I20260928 05:17:37.103507 4989 raft_consensus.cc:2781] T 416c2ab10d004761ab01d2b6854b4a42 P 5f03f228a75347008bdf55802a49afca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 05:17:37.103639 4989 raft_consensus.cc:482] T 416c2ab10d004761ab01d2b6854b4a42 P 5f03f228a75347008bdf55802a49afca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 05:17:37.103729 4989 raft_consensus.cc:3037] T 416c2ab10d004761ab01d2b6854b4a42 P 5f03f228a75347008bdf55802a49afca [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:17:37.105823 4989 raft_consensus.cc:504] T 416c2ab10d004761ab01d2b6854b4a42 P 5f03f228a75347008bdf55802a49afca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f03f228a75347008bdf55802a49afca" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40591 } } peers { permanent_uuid: "1dbd21d244bc4eacb6d7ac79c32fbd51" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40579 } } peers { permanent_uuid: "f2d94c8bcba3402d906415ca372c0523" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41471 } }
I20260928 05:17:37.106339 4989 leader_election.cc:288] T 416c2ab10d004761ab01d2b6854b4a42 P 5f03f228a75347008bdf55802a49afca [CANDIDATE]: Term 1 election: Requested vote from peers 1dbd21d244bc4eacb6d7ac79c32fbd51 (127.0.97.2:40579), f2d94c8bcba3402d906415ca372c0523 (127.0.97.1:41471)
I20260928 05:17:37.106681 4787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "416c2ab10d004761ab01d2b6854b4a42" candidate_uuid: "5f03f228a75347008bdf55802a49afca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dbd21d244bc4eacb6d7ac79c32fbd51"
I20260928 05:17:37.106770 4654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "416c2ab10d004761ab01d2b6854b4a42" candidate_uuid: "5f03f228a75347008bdf55802a49afca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2d94c8bcba3402d906415ca372c0523"
I20260928 05:17:37.106843 4787 raft_consensus.cc:3037] T 416c2ab10d004761ab01d2b6854b4a42 P 1dbd21d244bc4eacb6d7ac79c32fbd51 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:17:37.106935 4654 raft_consensus.cc:3037] T 416c2ab10d004761ab01d2b6854b4a42 P f2d94c8bcba3402d906415ca372c0523 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:17:37.108839 4787 raft_consensus.cc:2454] T 416c2ab10d004761ab01d2b6854b4a42 P 1dbd21d244bc4eacb6d7ac79c32fbd51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f03f228a75347008bdf55802a49afca in term 1.
I20260928 05:17:37.109021 4654 raft_consensus.cc:2454] T 416c2ab10d004761ab01d2b6854b4a42 P f2d94c8bcba3402d906415ca372c0523 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f03f228a75347008bdf55802a49afca in term 1.
I20260928 05:17:37.109202 4853 leader_election.cc:302] T 416c2ab10d004761ab01d2b6854b4a42 P 5f03f228a75347008bdf55802a49afca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dbd21d244bc4eacb6d7ac79c32fbd51, 5f03f228a75347008bdf55802a49afca; no voters:
I20260928 05:17:37.109463 4989 raft_consensus.cc:2781] T 416c2ab10d004761ab01d2b6854b4a42 P 5f03f228a75347008bdf55802a49afca [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:17:37.109848 4989 raft_consensus.cc:686] T 416c2ab10d004761ab01d2b6854b4a42 P 5f03f228a75347008bdf55802a49afca [term 1 LEADER]: Becoming Leader. State: Replica: 5f03f228a75347008bdf55802a49afca, State: Running, Role: LEADER
I20260928 05:17:37.110256 4989 consensus_queue.cc:237] T 416c2ab10d004761ab01d2b6854b4a42 P 5f03f228a75347008bdf55802a49afca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f03f228a75347008bdf55802a49afca" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40591 } } peers { permanent_uuid: "1dbd21d244bc4eacb6d7ac79c32fbd51" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40579 } } peers { permanent_uuid: "f2d94c8bcba3402d906415ca372c0523" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41471 } }
I20260928 05:17:37.114166 4509 catalog_manager.cc:5495] T 416c2ab10d004761ab01d2b6854b4a42 P 5f03f228a75347008bdf55802a49afca reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f03f228a75347008bdf55802a49afca (127.0.97.3). New cstate: current_term: 1 leader_uuid: "5f03f228a75347008bdf55802a49afca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f03f228a75347008bdf55802a49afca" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dbd21d244bc4eacb6d7ac79c32fbd51" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f2d94c8bcba3402d906415ca372c0523" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41471 } health_report { overall_health: UNKNOWN } } }
W20260928 05:17:37.147511 4833 tablet.cc:2369] T 416c2ab10d004761ab01d2b6854b4a42 P 1dbd21d244bc4eacb6d7ac79c32fbd51: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 05:17:37.149318 4977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:37.144378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35452 (local address 127.0.97.3:40591)
0928 05:17:37.144578 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:37.144585 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:37.144663 (+ 78us) server_negotiation.cc:407] Connection header received
0928 05:17:37.144823 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:37.144826 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:37.144939 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:37.145052 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:17:37.145341 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:37.146033 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:37.147246 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:37.147672 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:37.147789 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:37.148268 (+ 479us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:17:37.148750 (+ 482us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:17:37.148893 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:37.149014 (+ 121us) server_negotiation.cc:299] Negotiation successful
0928 05:17:37.149089 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 05:17:37.156769 4787 raft_consensus.cc:1261] T 416c2ab10d004761ab01d2b6854b4a42 P 1dbd21d244bc4eacb6d7ac79c32fbd51 [term 1 FOLLOWER]: Refusing update from remote peer 5f03f228a75347008bdf55802a49afca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:17:37.156769 4654 raft_consensus.cc:1261] T 416c2ab10d004761ab01d2b6854b4a42 P f2d94c8bcba3402d906415ca372c0523 [term 1 FOLLOWER]: Refusing update from remote peer 5f03f228a75347008bdf55802a49afca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:17:37.157675 4992 consensus_queue.cc:1035] T 416c2ab10d004761ab01d2b6854b4a42 P 5f03f228a75347008bdf55802a49afca [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dbd21d244bc4eacb6d7ac79c32fbd51" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:17:37.158039 4989 consensus_queue.cc:1035] T 416c2ab10d004761ab01d2b6854b4a42 P 5f03f228a75347008bdf55802a49afca [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2d94c8bcba3402d906415ca372c0523" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:17:37.167688 4995 mvcc.cc:205] Tried to move back new op lower bound from 7334185603704418304 to 7334185603526823936. Current Snapshot: MvccSnapshot[applied={T|T < 7334185603704418304}]
I20260928 05:17:37.168538 4997 mvcc.cc:205] Tried to move back new op lower bound from 7334185603704418304 to 7334185603526823936. Current Snapshot: MvccSnapshot[applied={T|T < 7334185603704418304}]
I20260928 05:17:37.174914 4996 mvcc.cc:205] Tried to move back new op lower bound from 7334185603704418304 to 7334185603526823936. Current Snapshot: MvccSnapshot[applied={T|T < 7334185603704418304}]
I20260928 05:17:37.182782 4510 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54740:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:17:37.182986 4510 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54740:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:17:37.186136 4510 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7f02e89feecb417597e5421d43aded8f: Sending DeleteTablet for 3 replicas of tablet 416c2ab10d004761ab01d2b6854b4a42
I20260928 05:17:37.187116 4899 tablet_service.cc:1514] Processing DeleteTablet for tablet 416c2ab10d004761ab01d2b6854b4a42 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:17:37 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35442
I20260928 05:17:37.187253 4767 tablet_service.cc:1514] Processing DeleteTablet for tablet 416c2ab10d004761ab01d2b6854b4a42 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:17:37 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60732
I20260928 05:17:37.187531 4634 tablet_service.cc:1514] Processing DeleteTablet for tablet 416c2ab10d004761ab01d2b6854b4a42 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:17:37 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54898
I20260928 05:17:37.189409 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 4572
I20260928 05:17:37.196885 5007 tablet_replica.cc:330] T 416c2ab10d004761ab01d2b6854b4a42 P f2d94c8bcba3402d906415ca372c0523: stopping tablet replica
I20260928 05:17:37.197472 5007 raft_consensus.cc:2229] T 416c2ab10d004761ab01d2b6854b4a42 P f2d94c8bcba3402d906415ca372c0523 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:17:37.198024 5007 raft_consensus.cc:2258] T 416c2ab10d004761ab01d2b6854b4a42 P f2d94c8bcba3402d906415ca372c0523 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:17:37.200255 5007 ts_tablet_manager.cc:1905] T 416c2ab10d004761ab01d2b6854b4a42 P f2d94c8bcba3402d906415ca372c0523: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:17:37.208413 5007 ts_tablet_manager.cc:1918] T 416c2ab10d004761ab01d2b6854b4a42 P f2d94c8bcba3402d906415ca372c0523: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:17:37.209022 5005 tablet_replica.cc:330] T 416c2ab10d004761ab01d2b6854b4a42 P 5f03f228a75347008bdf55802a49afca: stopping tablet replica
I20260928 05:17:37.209386 5005 raft_consensus.cc:2229] T 416c2ab10d004761ab01d2b6854b4a42 P 5f03f228a75347008bdf55802a49afca [term 1 LEADER]: Raft consensus shutting down.
I20260928 05:17:37.209909 5005 raft_consensus.cc:2258] T 416c2ab10d004761ab01d2b6854b4a42 P 5f03f228a75347008bdf55802a49afca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:17:37.210613 5006 tablet_replica.cc:330] T 416c2ab10d004761ab01d2b6854b4a42 P 1dbd21d244bc4eacb6d7ac79c32fbd51: stopping tablet replica
I20260928 05:17:37.211051 5006 raft_consensus.cc:2229] T 416c2ab10d004761ab01d2b6854b4a42 P 1dbd21d244bc4eacb6d7ac79c32fbd51 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:17:37.211522 5006 raft_consensus.cc:2258] T 416c2ab10d004761ab01d2b6854b4a42 P 1dbd21d244bc4eacb6d7ac79c32fbd51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:17:37.211908 5007 log.cc:1196] T 416c2ab10d004761ab01d2b6854b4a42 P f2d94c8bcba3402d906415ca372c0523: Deleting WAL directory at /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572622576160-388-0/minicluster-data/ts-0/wal/wals/416c2ab10d004761ab01d2b6854b4a42
I20260928 05:17:37.211989 5005 ts_tablet_manager.cc:1905] T 416c2ab10d004761ab01d2b6854b4a42 P 5f03f228a75347008bdf55802a49afca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:17:37.212424 5007 ts_tablet_manager.cc:1939] T 416c2ab10d004761ab01d2b6854b4a42 P f2d94c8bcba3402d906415ca372c0523: Deleting consensus metadata
I20260928 05:17:37.214195 4495 catalog_manager.cc:4841] TS f2d94c8bcba3402d906415ca372c0523 (127.0.97.1:41471): tablet 416c2ab10d004761ab01d2b6854b4a42 (table test-table [id=38e6ae17052040f381227d9b0b1c5372]) successfully deleted
I20260928 05:17:37.215401 5006 ts_tablet_manager.cc:1905] T 416c2ab10d004761ab01d2b6854b4a42 P 1dbd21d244bc4eacb6d7ac79c32fbd51: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:17:37.218875 5005 ts_tablet_manager.cc:1918] T 416c2ab10d004761ab01d2b6854b4a42 P 5f03f228a75347008bdf55802a49afca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:17:37.219022 5005 log.cc:1196] T 416c2ab10d004761ab01d2b6854b4a42 P 5f03f228a75347008bdf55802a49afca: Deleting WAL directory at /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572622576160-388-0/minicluster-data/ts-2/wal/wals/416c2ab10d004761ab01d2b6854b4a42
I20260928 05:17:37.219509 5005 ts_tablet_manager.cc:1939] T 416c2ab10d004761ab01d2b6854b4a42 P 5f03f228a75347008bdf55802a49afca: Deleting consensus metadata
Sep 28 05:17:37 dist-test-slave-whsn krb5kdc[4466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572654, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260928 05:17:37.220985 4494 catalog_manager.cc:4841] TS 5f03f228a75347008bdf55802a49afca (127.0.97.3:40591): tablet 416c2ab10d004761ab01d2b6854b4a42 (table test-table [id=38e6ae17052040f381227d9b0b1c5372]) successfully deleted
I20260928 05:17:37.221855 5006 ts_tablet_manager.cc:1918] T 416c2ab10d004761ab01d2b6854b4a42 P 1dbd21d244bc4eacb6d7ac79c32fbd51: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:17:37.221995 5006 log.cc:1196] T 416c2ab10d004761ab01d2b6854b4a42 P 1dbd21d244bc4eacb6d7ac79c32fbd51: Deleting WAL directory at /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572622576160-388-0/minicluster-data/ts-1/wal/wals/416c2ab10d004761ab01d2b6854b4a42
I20260928 05:17:37.222376 5006 ts_tablet_manager.cc:1939] T 416c2ab10d004761ab01d2b6854b4a42 P 1dbd21d244bc4eacb6d7ac79c32fbd51: Deleting consensus metadata
I20260928 05:17:37.223701 4495 catalog_manager.cc:4841] TS 1dbd21d244bc4eacb6d7ac79c32fbd51 (127.0.97.2:40579): tablet 416c2ab10d004761ab01d2b6854b4a42 (table test-table [id=38e6ae17052040f381227d9b0b1c5372]) successfully deleted
I20260928 05:17:37.228286 4980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:37.190604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54912 (local address 127.0.97.1:41471)
0928 05:17:37.190803 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:37.190810 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:37.200837 (+ 10027us) server_negotiation.cc:407] Connection header received
0928 05:17:37.201095 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:37.201102 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:37.201194 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:37.201305 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:37.203278 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:37.204504 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:37.217004 (+ 12500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:37.217658 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:37.217782 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:37.221769 (+ 3987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:37.221783 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:37.221800 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:37.221843 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:37.225528 (+ 3685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:37.226177 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:37.226186 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:37.226190 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:37.226342 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:37.226736 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:37.226742 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:37.226745 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:37.227229 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:37.227546 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:37.227709 (+ 163us) server_negotiation.cc:299] Negotiation successful
0928 05:17:37.227920 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 05:17:37.228998 4694 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:17:37.336974 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 4572
I20260928 05:17:37.356696 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 4705
Sep 28 05:17:37 dist-test-slave-whsn krb5kdc[4466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572654, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260928 05:17:37.372252 4979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:37.357484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60746 (local address 127.0.97.2:40579)
0928 05:17:37.357669 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:37.357676 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:37.357696 (+ 20us) server_negotiation.cc:407] Connection header received
0928 05:17:37.357761 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:37.357767 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:37.357869 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:37.357976 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:37.359022 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:37.360123 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:37.361356 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:37.361978 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:37.362113 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:37.365940 (+ 3827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:37.365959 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:37.365979 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:37.366028 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:37.369746 (+ 3718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:37.370359 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:37.370368 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:37.370372 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:37.370515 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:37.370821 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:37.370826 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:37.370829 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:37.371264 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:37.371546 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:37.371695 (+ 149us) server_negotiation.cc:299] Negotiation successful
0928 05:17:37.371895 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 05:17:37.372928 4827 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:17:37.468051 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 4705
I20260928 05:17:37.487232 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 4837
Sep 28 05:17:37 dist-test-slave-whsn krb5kdc[4466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572654, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260928 05:17:37.505803 4977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:37.487972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35458 (local address 127.0.97.3:40591)
0928 05:17:37.488141 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:37.488147 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:37.488165 (+ 18us) server_negotiation.cc:407] Connection header received
0928 05:17:37.488254 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:37.488259 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:37.488329 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:37.488412 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:37.489597 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:37.490374 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:37.491596 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:37.492044 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:37.492169 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:37.500365 (+ 8196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:37.500376 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:37.500394 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:37.500430 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:37.503188 (+ 2758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:37.503950 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:37.503957 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:37.503959 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:37.504097 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:37.504476 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:37.504479 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:37.504481 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:37.504809 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:37.505075 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:37.505347 (+ 272us) server_negotiation.cc:299] Negotiation successful
0928 05:17:37.505506 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 05:17:37.506525 4959 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:17:37.621311 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 4837
I20260928 05:17:37.635970 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 4479
I20260928 05:17:37.636987 4540 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:17:37.739243 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 4479
2026-09-28T05:17:37Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (3882 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-28T05:17:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:17:37Z Disabled control of system clock
I20260928 05:17:37.794598 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572622576160-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572622576160-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572622576160-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43437
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:46591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43437
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 05:17:38.088927 5016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:17:38.089187 5016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:17:38.089309 5016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:17:38.096150 5016 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:17:38.096248 5016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:17:38.096310 5016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:17:38.096379 5016 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:17:38.096443 5016 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:17:38.105222 5016 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572622576160-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572622576160-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43437
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572622576160-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:17:38.106335 5016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:17:38.108031 5016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:17:38.118901 5022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:38.119051 5021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:38.119019 5024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:17:38.119488 5016 server_base.cc:1034] running on GCE node
I20260928 05:17:38.120142 5016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:17:38.121426 5016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:17:38.122629 5016 hybrid_clock.cc:648] HybridClock initialized: now 1790572658122573 us; error 44 us; skew 500 ppm
I20260928 05:17:38.122937 5016 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 05:17:38.124981 5016 webserver.cc:460] Webserver started at http://127.0.97.62:45933/ using document root <none> and password file <none>
I20260928 05:17:38.125586 5016 fs_manager.cc:359] Metadata directory not provided
I20260928 05:17:38.125689 5016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:17:38.125996 5016 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:17:38.128762 5016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572622576160-388-0/minicluster-data/master-0/data/instance:
uuid: "64be312980ad41e1949184b33c4537c6"
format_stamp: "Formatted at 2026-09-28 05:17:38 on dist-test-slave-whsn"
server_key: "26a7adfcbd62727c58bbf954382159a7"
server_key_iv: "a1e1a3071601d166f1f41b7fded498aa"
server_key_version: "encryptionkey@0"
I20260928 05:17:38.129535 5016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572622576160-388-0/minicluster-data/master-0/wal/instance:
uuid: "64be312980ad41e1949184b33c4537c6"
format_stamp: "Formatted at 2026-09-28 05:17:38 on dist-test-slave-whsn"
server_key: "26a7adfcbd62727c58bbf954382159a7"
server_key_iv: "a1e1a3071601d166f1f41b7fded498aa"
server_key_version: "encryptionkey@0"
I20260928 05:17:38.134191 5016 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 05:17:38.137290 5030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:38.138471 5016 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260928 05:17:38.138621 5016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572622576160-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572622576160-388-0/minicluster-data/master-0/wal
uuid: "64be312980ad41e1949184b33c4537c6"
format_stamp: "Formatted at 2026-09-28 05:17:38 on dist-test-slave-whsn"
server_key: "26a7adfcbd62727c58bbf954382159a7"
server_key_iv: "a1e1a3071601d166f1f41b7fded498aa"
server_key_version: "encryptionkey@0"
I20260928 05:17:38.138800 5016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572622576160-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572622576160-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572622576160-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:17:38.144039 5016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:17:38.144721 5016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:17:38.145018 5016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:17:38.160624 5016 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43437
I20260928 05:17:38.160650 5081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43437 every 8 connection(s)
I20260928 05:17:38.161752 5016 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:17:38.165001 5082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:17:38.167953 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 5016
I20260928 05:17:38.168144 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572622576160-388-0/minicluster-data/master-0/wal/instance
I20260928 05:17:38.168697 388 external_mini_cluster.cc:1428] Setting key 0c8d87d6974858567291d37e120b738d
I20260928 05:17:38.175951 5085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:38.170462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47328 (local address 127.0.97.62:43437)
0928 05:17:38.171315 (+ 853us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:38.171325 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:38.171363 (+ 38us) server_negotiation.cc:407] Connection header received
0928 05:17:38.172809 (+ 1446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:38.172833 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:38.173262 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:38.173569 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:38.174547 (+ 978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:38.174552 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:38.174564 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:17:38.174580 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:38.174646 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:17:38.174662 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:38.174812 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:38.174919 (+ 107us) server_negotiation.cc:299] Negotiation successful
0928 05:17:38.175133 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":303,"threads_started":1}
I20260928 05:17:38.176442 5082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64be312980ad41e1949184b33c4537c6: Bootstrap starting.
I20260928 05:17:38.179145 5082 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 64be312980ad41e1949184b33c4537c6: Neither blocks nor log segments found. Creating new log.
I20260928 05:17:38.180020 5082 log.cc:824] T 00000000000000000000000000000000 P 64be312980ad41e1949184b33c4537c6: Log is configured to *not* fsync() on all Append() calls
I20260928 05:17:38.182675 5082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64be312980ad41e1949184b33c4537c6: No bootstrap required, opened a new log
I20260928 05:17:38.189620 5082 raft_consensus.cc:348] T 00000000000000000000000000000000 P 64be312980ad41e1949184b33c4537c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64be312980ad41e1949184b33c4537c6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43437 } }
I20260928 05:17:38.189910 5082 raft_consensus.cc:374] T 00000000000000000000000000000000 P 64be312980ad41e1949184b33c4537c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:17:38.190006 5082 raft_consensus.cc:729] T 00000000000000000000000000000000 P 64be312980ad41e1949184b33c4537c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64be312980ad41e1949184b33c4537c6, State: Initialized, Role: FOLLOWER
I20260928 05:17:38.190501 5082 consensus_queue.cc:260] T 00000000000000000000000000000000 P 64be312980ad41e1949184b33c4537c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64be312980ad41e1949184b33c4537c6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43437 } }
I20260928 05:17:38.190698 5082 raft_consensus.cc:388] T 00000000000000000000000000000000 P 64be312980ad41e1949184b33c4537c6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:17:38.190815 5082 raft_consensus.cc:482] T 00000000000000000000000000000000 P 64be312980ad41e1949184b33c4537c6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:17:38.190944 5082 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 64be312980ad41e1949184b33c4537c6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:17:38.192840 5082 raft_consensus.cc:504] T 00000000000000000000000000000000 P 64be312980ad41e1949184b33c4537c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64be312980ad41e1949184b33c4537c6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43437 } }
I20260928 05:17:38.193310 5082 leader_election.cc:302] T 00000000000000000000000000000000 P 64be312980ad41e1949184b33c4537c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 64be312980ad41e1949184b33c4537c6; no voters:
I20260928 05:17:38.193781 5082 leader_election.cc:288] T 00000000000000000000000000000000 P 64be312980ad41e1949184b33c4537c6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:17:38.193878 5087 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 64be312980ad41e1949184b33c4537c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:17:38.194521 5087 raft_consensus.cc:686] T 00000000000000000000000000000000 P 64be312980ad41e1949184b33c4537c6 [term 1 LEADER]: Becoming Leader. State: Replica: 64be312980ad41e1949184b33c4537c6, State: Running, Role: LEADER
I20260928 05:17:38.195029 5087 consensus_queue.cc:237] T 00000000000000000000000000000000 P 64be312980ad41e1949184b33c4537c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64be312980ad41e1949184b33c4537c6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43437 } }
I20260928 05:17:38.195531 5082 sys_catalog.cc:564] T 00000000000000000000000000000000 P 64be312980ad41e1949184b33c4537c6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:17:38.197518 5089 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64be312980ad41e1949184b33c4537c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 64be312980ad41e1949184b33c4537c6. Latest consensus state: current_term: 1 leader_uuid: "64be312980ad41e1949184b33c4537c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64be312980ad41e1949184b33c4537c6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43437 } } }
I20260928 05:17:38.197492 5088 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64be312980ad41e1949184b33c4537c6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "64be312980ad41e1949184b33c4537c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64be312980ad41e1949184b33c4537c6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43437 } } }
I20260928 05:17:38.197863 5089 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64be312980ad41e1949184b33c4537c6 [sys.catalog]: This master's current role is: LEADER
I20260928 05:17:38.197861 5088 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64be312980ad41e1949184b33c4537c6 [sys.catalog]: This master's current role is: LEADER
W20260928 05:17:38.209872 5103 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 64be312980ad41e1949184b33c4537c6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 05:17:38.209968 5103 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 05:17:38.210078 5094 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:17:38.211484 5094 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:17:38.218729 5094 catalog_manager.cc:1347] Generated new cluster ID: 4d6d02c004b44eddaac75e4a8d621c62
I20260928 05:17:38.218816 5094 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:17:38.235637 5094 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:17:38.236058 5094 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:17:38.245906 5094 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 64be312980ad41e1949184b33c4537c6: Generated new TSK 0
I20260928 05:17:38.246582 5094 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 05:17:38.258793 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572622576160-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572622576160-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572622576160-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572622576160-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43437
--builtin_ntp_servers=127.0.97.20:46591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 05:17:38.562647 5106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:17:38.562885 5106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:17:38.563009 5106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:17:38.569893 5106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:17:38.569976 5106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:17:38.570109 5106 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 05:17:38.578673 5106 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572622576160-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572622576160-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572622576160-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43437
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572622576160-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:17:38.579756 5106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:17:38.581418 5106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:17:38.592994 5112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:38.593087 5111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:38.593063 5114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:17:38.593809 5106 server_base.cc:1034] running on GCE node
I20260928 05:17:38.594329 5106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:17:38.595184 5106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:17:38.596356 5106 hybrid_clock.cc:648] HybridClock initialized: now 1790572658596304 us; error 48 us; skew 500 ppm
I20260928 05:17:38.596627 5106 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 05:17:38.598604 5106 webserver.cc:460] Webserver started at http://127.0.97.1:40145/ using document root <none> and password file <none>
I20260928 05:17:38.599195 5106 fs_manager.cc:359] Metadata directory not provided
I20260928 05:17:38.599278 5106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:17:38.599567 5106 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:17:38.602165 5106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572622576160-388-0/minicluster-data/ts-0/data/instance:
uuid: "eca28ad6d2534d48b3e71750317a4dc3"
format_stamp: "Formatted at 2026-09-28 05:17:38 on dist-test-slave-whsn"
server_key: "0cbc932a683650e01f6fe2a4ade67482"
server_key_iv: "fcb435b02b971678b961b89698960bd2"
server_key_version: "encryptionkey@0"
I20260928 05:17:38.602860 5106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572622576160-388-0/minicluster-data/ts-0/wal/instance:
uuid: "eca28ad6d2534d48b3e71750317a4dc3"
format_stamp: "Formatted at 2026-09-28 05:17:38 on dist-test-slave-whsn"
server_key: "0cbc932a683650e01f6fe2a4ade67482"
server_key_iv: "fcb435b02b971678b961b89698960bd2"
server_key_version: "encryptionkey@0"
I20260928 05:17:38.607610 5106 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260928 05:17:38.610664 5120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:38.611814 5106 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260928 05:17:38.611940 5106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572622576160-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572622576160-388-0/minicluster-data/ts-0/wal
uuid: "eca28ad6d2534d48b3e71750317a4dc3"
format_stamp: "Formatted at 2026-09-28 05:17:38 on dist-test-slave-whsn"
server_key: "0cbc932a683650e01f6fe2a4ade67482"
server_key_iv: "fcb435b02b971678b961b89698960bd2"
server_key_version: "encryptionkey@0"
I20260928 05:17:38.612128 5106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572622576160-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572622576160-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572622576160-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:17:38.617240 5106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:17:38.617914 5106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:17:38.618139 5106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:17:38.618922 5106 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:17:38.620154 5106 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:17:38.620244 5106 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:38.620345 5106 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:17:38.620410 5106 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:38.648666 5106 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38973
I20260928 05:17:38.648707 5232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38973 every 8 connection(s)
I20260928 05:17:38.649896 5106 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572622576160-388-0/minicluster-data/ts-0/data/info.pb
I20260928 05:17:38.655745 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 5106
I20260928 05:17:38.655920 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572622576160-388-0/minicluster-data/ts-0/wal/instance
I20260928 05:17:38.656584 388 external_mini_cluster.cc:1428] Setting key 2696b900421c7aca3545c88e87cc5ea8
I20260928 05:17:38.657231 5085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:38.652159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60893 (local address 127.0.97.62:43437)
0928 05:17:38.652350 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:38.652356 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:38.653671 (+ 1315us) server_negotiation.cc:407] Connection header received
0928 05:17:38.655196 (+ 1525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:38.655200 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:38.655247 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:38.655331 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:38.656566 (+ 1235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:38.656572 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:38.656577 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:17:38.656595 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:38.656633 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:17:38.656649 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:38.656766 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:38.656928 (+ 162us) server_negotiation.cc:299] Negotiation successful
0928 05:17:38.656985 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 05:17:38.658136 5235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:38.652660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43437 (local address 127.0.97.1:60893)
0928 05:17:38.653535 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:38.653574 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:38.654981 (+ 1407us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:38.655516 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:38.655523 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:38.655610 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:17:38.656257 (+ 647us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 05:17:38.656259 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:17:38.656295 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 05:17:38.656375 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:17:38.656738 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:17:38.656747 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:17:38.656752 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 05:17:38.656987 (+ 235us) client_negotiation.cc:240] Negotiation successful
0928 05:17:38.657185 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":281,"threads_started":1}
I20260928 05:17:38.659291 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572622576160-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572622576160-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572622576160-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572622576160-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43437
--builtin_ntp_servers=127.0.97.20:46591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260928 05:17:38.660957 5233 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43437
I20260928 05:17:38.661235 5233 heartbeater.cc:464] Registering TS with master...
I20260928 05:17:38.661821 5233 heartbeater.cc:511] Master 127.0.97.62:43437 requested a full tablet report, sending...
I20260928 05:17:38.663890 5047 ts_manager.cc:194] Registered new tserver with Master: eca28ad6d2534d48b3e71750317a4dc3 (127.0.97.1:38973)
W20260928 05:17:38.952745 5236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:17:38.953014 5236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:17:38.953140 5236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:17:38.959887 5236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:17:38.959972 5236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:17:38.960078 5236 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 05:17:38.968438 5236 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572622576160-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572622576160-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572622576160-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43437
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572622576160-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:17:38.969509 5236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:17:38.971076 5236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:17:38.982538 5244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:38.982604 5241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:38.982558 5242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:17:38.983316 5236 server_base.cc:1034] running on GCE node
I20260928 05:17:38.983872 5236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:17:38.984692 5236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:17:38.985874 5236 hybrid_clock.cc:648] HybridClock initialized: now 1790572658985824 us; error 41 us; skew 500 ppm
I20260928 05:17:38.986150 5236 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 05:17:38.988111 5236 webserver.cc:460] Webserver started at http://127.0.97.2:36345/ using document root <none> and password file <none>
I20260928 05:17:38.988706 5236 fs_manager.cc:359] Metadata directory not provided
I20260928 05:17:38.988796 5236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:17:38.989080 5236 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:17:38.991669 5236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572622576160-388-0/minicluster-data/ts-1/data/instance:
uuid: "04478c3d0d6a44bc8fc7c61b5947428b"
format_stamp: "Formatted at 2026-09-28 05:17:38 on dist-test-slave-whsn"
server_key: "8e64d1da3d17f3c05adcfe0be382296d"
server_key_iv: "f2f9ef33a3159c74cfc6ab7ddb0e85e7"
server_key_version: "encryptionkey@0"
I20260928 05:17:38.992379 5236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572622576160-388-0/minicluster-data/ts-1/wal/instance:
uuid: "04478c3d0d6a44bc8fc7c61b5947428b"
format_stamp: "Formatted at 2026-09-28 05:17:38 on dist-test-slave-whsn"
server_key: "8e64d1da3d17f3c05adcfe0be382296d"
server_key_iv: "f2f9ef33a3159c74cfc6ab7ddb0e85e7"
server_key_version: "encryptionkey@0"
I20260928 05:17:38.997097 5236 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260928 05:17:39.000089 5250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:39.001277 5236 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 05:17:39.001415 5236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572622576160-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572622576160-388-0/minicluster-data/ts-1/wal
uuid: "04478c3d0d6a44bc8fc7c61b5947428b"
format_stamp: "Formatted at 2026-09-28 05:17:38 on dist-test-slave-whsn"
server_key: "8e64d1da3d17f3c05adcfe0be382296d"
server_key_iv: "f2f9ef33a3159c74cfc6ab7ddb0e85e7"
server_key_version: "encryptionkey@0"
I20260928 05:17:39.001608 5236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572622576160-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572622576160-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572622576160-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:17:39.006763 5236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:17:39.007436 5236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:17:39.007664 5236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:17:39.008442 5236 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:17:39.009693 5236 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:17:39.009783 5236 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:39.009883 5236 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:17:39.009948 5236 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:39.037801 5236 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44387
I20260928 05:17:39.037850 5362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44387 every 8 connection(s)
I20260928 05:17:39.038985 5236 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572622576160-388-0/minicluster-data/ts-1/data/info.pb
I20260928 05:17:39.044679 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 5236
I20260928 05:17:39.044888 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572622576160-388-0/minicluster-data/ts-1/wal/instance
I20260928 05:17:39.045365 388 external_mini_cluster.cc:1428] Setting key a44efbf0173dd9ea70f6d421c9a80347
I20260928 05:17:39.046427 5085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:39.041227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53875 (local address 127.0.97.62:43437)
0928 05:17:39.041382 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:39.041386 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:39.042762 (+ 1376us) server_negotiation.cc:407] Connection header received
0928 05:17:39.044292 (+ 1530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:39.044296 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:39.044343 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:39.044424 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:39.045788 (+ 1364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:39.045792 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:39.045795 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:17:39.045809 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:39.045838 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:17:39.045849 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:39.045959 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:39.046122 (+ 163us) server_negotiation.cc:299] Negotiation successful
0928 05:17:39.046180 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 05:17:39.047355 5365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:39.041706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43437 (local address 127.0.97.2:53875)
0928 05:17:39.042636 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:39.042670 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:39.044087 (+ 1417us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:39.044682 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:39.044693 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:39.044807 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:17:39.045472 (+ 665us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 05:17:39.045476 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:17:39.045522 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 05:17:39.045630 (+ 108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:17:39.045949 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:17:39.045957 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:17:39.045962 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 05:17:39.046179 (+ 217us) client_negotiation.cc:240] Negotiation successful
0928 05:17:39.046400 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":281,"threads_started":1}
I20260928 05:17:39.048208 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572622576160-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572622576160-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572622576160-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572622576160-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43437
--builtin_ntp_servers=127.0.97.20:46591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260928 05:17:39.048545 5363 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43437
I20260928 05:17:39.048787 5363 heartbeater.cc:464] Registering TS with master...
I20260928 05:17:39.049295 5363 heartbeater.cc:511] Master 127.0.97.62:43437 requested a full tablet report, sending...
I20260928 05:17:39.050498 5047 ts_manager.cc:194] Registered new tserver with Master: 04478c3d0d6a44bc8fc7c61b5947428b (127.0.97.2:44387)
W20260928 05:17:39.354418 5366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:17:39.354671 5366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:17:39.354802 5366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:17:39.361757 5366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:17:39.361846 5366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:17:39.362018 5366 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 05:17:39.370689 5366 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572622576160-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572622576160-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572622576160-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43437
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572622576160-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:17:39.371799 5366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:17:39.373502 5366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:17:39.385670 5372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:39.385638 5374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:39.386003 5371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:17:39.386720 5366 server_base.cc:1034] running on GCE node
I20260928 05:17:39.387239 5366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:17:39.388103 5366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:17:39.389299 5366 hybrid_clock.cc:648] HybridClock initialized: now 1790572659389259 us; error 34 us; skew 500 ppm
I20260928 05:17:39.389580 5366 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 05:17:39.391609 5366 webserver.cc:460] Webserver started at http://127.0.97.3:45695/ using document root <none> and password file <none>
I20260928 05:17:39.392212 5366 fs_manager.cc:359] Metadata directory not provided
I20260928 05:17:39.392292 5366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:17:39.392527 5366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:17:39.395156 5366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572622576160-388-0/minicluster-data/ts-2/data/instance:
uuid: "0d866a0915e643b689492d22b6a9a1ed"
format_stamp: "Formatted at 2026-09-28 05:17:39 on dist-test-slave-whsn"
server_key: "53cb98bbf79d5a1c2ebad4c6515ca8ec"
server_key_iv: "6d7b5bd7b0a709f5f6091db0a8c08ca7"
server_key_version: "encryptionkey@0"
I20260928 05:17:39.395901 5366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572622576160-388-0/minicluster-data/ts-2/wal/instance:
uuid: "0d866a0915e643b689492d22b6a9a1ed"
format_stamp: "Formatted at 2026-09-28 05:17:39 on dist-test-slave-whsn"
server_key: "53cb98bbf79d5a1c2ebad4c6515ca8ec"
server_key_iv: "6d7b5bd7b0a709f5f6091db0a8c08ca7"
server_key_version: "encryptionkey@0"
I20260928 05:17:39.400532 5366 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260928 05:17:39.403709 5380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:39.405047 5366 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:17:39.405182 5366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572622576160-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572622576160-388-0/minicluster-data/ts-2/wal
uuid: "0d866a0915e643b689492d22b6a9a1ed"
format_stamp: "Formatted at 2026-09-28 05:17:39 on dist-test-slave-whsn"
server_key: "53cb98bbf79d5a1c2ebad4c6515ca8ec"
server_key_iv: "6d7b5bd7b0a709f5f6091db0a8c08ca7"
server_key_version: "encryptionkey@0"
I20260928 05:17:39.405395 5366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572622576160-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572622576160-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572622576160-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:17:39.410562 5366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:17:39.411248 5366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:17:39.411466 5366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:17:39.412294 5366 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:17:39.413578 5366 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:17:39.413663 5366 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:39.413761 5366 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:17:39.413828 5366 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:39.441470 5366 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:32947
I20260928 05:17:39.441532 5492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:32947 every 8 connection(s)
I20260928 05:17:39.442687 5366 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572622576160-388-0/minicluster-data/ts-2/data/info.pb
I20260928 05:17:39.449823 5085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:39.445059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36157 (local address 127.0.97.62:43437)
0928 05:17:39.445266 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:39.445272 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:39.446571 (+ 1299us) server_negotiation.cc:407] Connection header received
0928 05:17:39.448064 (+ 1493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:39.448068 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:39.448114 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:39.448195 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:39.449332 (+ 1137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:39.449335 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:39.449338 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:17:39.449350 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:39.449376 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:17:39.449386 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:39.449473 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:39.449600 (+ 127us) server_negotiation.cc:299] Negotiation successful
0928 05:17:39.449643 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 05:17:39.450630 5495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:39.445576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43437 (local address 127.0.97.3:36157)
0928 05:17:39.446429 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:39.446462 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:39.447846 (+ 1384us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:39.448366 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:39.448373 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:39.448466 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:17:39.449085 (+ 619us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 05:17:39.449093 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:17:39.449127 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 05:17:39.449203 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:17:39.449455 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:17:39.449460 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:17:39.449465 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 05:17:39.449654 (+ 189us) client_negotiation.cc:240] Negotiation successful
0928 05:17:39.449823 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":261,"threads_started":1}
I20260928 05:17:39.451813 5493 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43437
I20260928 05:17:39.452033 5493 heartbeater.cc:464] Registering TS with master...
I20260928 05:17:39.452513 5493 heartbeater.cc:511] Master 127.0.97.62:43437 requested a full tablet report, sending...
I20260928 05:17:39.452965 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 5366
I20260928 05:17:39.453155 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572622576160-388-0/minicluster-data/ts-2/wal/instance
I20260928 05:17:39.453583 388 external_mini_cluster.cc:1428] Setting key 79e1b291ddb770360490feec7b7682c6
I20260928 05:17:39.453920 5047 ts_manager.cc:194] Registered new tserver with Master: 0d866a0915e643b689492d22b6a9a1ed (127.0.97.3:32947)
I20260928 05:17:39.456380 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 05:17:39.462765 5085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:39.460398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47344 (local address 127.0.97.62:43437)
0928 05:17:39.460547 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:39.460554 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:39.460928 (+ 374us) server_negotiation.cc:407] Connection header received
0928 05:17:39.461114 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:39.461120 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:39.461170 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:39.461246 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:39.462050 (+ 804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:39.462055 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:39.462060 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:17:39.462077 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:39.462106 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:17:39.462117 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:39.462213 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:39.462313 (+ 100us) server_negotiation.cc:299] Negotiation successful
0928 05:17:39.462366 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 05:17:39.466207 5046 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47344:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 05:17:39.472512 5046 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 05:17:39.488214 5506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:39.483207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38973 (local address 127.0.0.1:60452)
0928 05:17:39.485251 (+ 2044us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:39.485264 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:39.485353 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:39.486550 (+ 1197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:39.486556 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:39.486620 (+ 64us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:17:39.487204 (+ 584us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 05:17:39.487207 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:17:39.487226 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 05:17:39.487286 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:17:39.487614 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:17:39.487623 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:17:39.487631 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 05:17:39.487763 (+ 132us) client_negotiation.cc:240] Negotiation successful
0928 05:17:39.487849 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1958,"thread_start_us":738,"threads_started":1}
I20260928 05:17:39.488960 5510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:39.483707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60452 (local address 127.0.97.1:38973)
0928 05:17:39.484138 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:39.484144 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:39.485353 (+ 1209us) server_negotiation.cc:407] Connection header received
0928 05:17:39.485473 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:39.485479 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:39.485662 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:39.485796 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:39.487426 (+ 1630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:39.487434 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:39.487448 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:17:39.487471 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:39.487514 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:17:39.487528 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:39.487636 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:39.487779 (+ 143us) server_negotiation.cc:299] Negotiation successful
0928 05:17:39.487864 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":350,"threads_started":1}
I20260928 05:17:39.491425 5507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:39.482536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37426 (local address 127.0.97.3:32947)
0928 05:17:39.489319 (+ 6783us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:39.489326 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:39.489356 (+ 30us) server_negotiation.cc:407] Connection header received
0928 05:17:39.489436 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:39.489443 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:39.489640 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:39.489776 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:39.490689 (+ 913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:39.490696 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:39.490708 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:17:39.490723 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:39.490770 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:17:39.490791 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:39.490944 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:39.490998 (+ 54us) server_negotiation.cc:299] Negotiation successful
0928 05:17:39.491084 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6628,"thread_start_us":6494,"threads_started":1}
I20260928 05:17:39.491461 5505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:39.482408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:32947 (local address 127.0.0.1:37426)
0928 05:17:39.483054 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:39.483094 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:39.483226 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:39.489808 (+ 6582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:39.489814 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:39.489840 (+ 26us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:17:39.490464 (+ 624us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 05:17:39.490468 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:17:39.490484 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 05:17:39.490535 (+ 51us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:17:39.490889 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:17:39.490893 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:17:39.490898 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 05:17:39.491005 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 05:17:39.491055 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":265,"threads_started":1}
I20260928 05:17:39.493371 5168 tablet_service.cc:1467] Processing CreateTablet for tablet d94779c029c54c4fa26fd8077624f676 (DEFAULT_TABLE table=test-table [id=6b4aac3d369d4785ae82b701b2174aaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:17:39.494787 5168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d94779c029c54c4fa26fd8077624f676. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:17:39.494802 5509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:39.482503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44387 (local address 127.0.0.1:46808)
0928 05:17:39.483650 (+ 1147us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:39.483663 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:39.483727 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:39.492461 (+ 8734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:39.492466 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:39.492485 (+ 19us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:17:39.493084 (+ 599us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 05:17:39.493088 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:17:39.493102 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 05:17:39.493139 (+ 37us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:17:39.494266 (+ 1127us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:17:39.494270 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:17:39.494276 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 05:17:39.494419 (+ 143us) client_negotiation.cc:240] Negotiation successful
0928 05:17:39.494464 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1038,"thread_start_us":2523,"threads_started":1}
I20260928 05:17:39.494828 5508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:39.483043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46808 (local address 127.0.97.2:44387)
0928 05:17:39.491984 (+ 8941us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:39.491992 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:39.492021 (+ 29us) server_negotiation.cc:407] Connection header received
0928 05:17:39.492086 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:39.492093 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:39.492271 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:39.492841 (+ 570us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:39.494051 (+ 1210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:39.494058 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:39.494070 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:17:39.494093 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:39.494138 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:17:39.494155 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:39.494311 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:39.494416 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 05:17:39.494499 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8791,"thread_start_us":8647,"threads_started":1}
I20260928 05:17:39.494647 5428 tablet_service.cc:1467] Processing CreateTablet for tablet d94779c029c54c4fa26fd8077624f676 (DEFAULT_TABLE table=test-table [id=6b4aac3d369d4785ae82b701b2174aaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:17:39.495827 5428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d94779c029c54c4fa26fd8077624f676. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:17:39.498119 5298 tablet_service.cc:1467] Processing CreateTablet for tablet d94779c029c54c4fa26fd8077624f676 (DEFAULT_TABLE table=test-table [id=6b4aac3d369d4785ae82b701b2174aaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:17:39.499629 5298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d94779c029c54c4fa26fd8077624f676. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:17:39.506161 5511 tablet_bootstrap.cc:492] T d94779c029c54c4fa26fd8077624f676 P eca28ad6d2534d48b3e71750317a4dc3: Bootstrap starting.
I20260928 05:17:39.509063 5511 tablet_bootstrap.cc:654] T d94779c029c54c4fa26fd8077624f676 P eca28ad6d2534d48b3e71750317a4dc3: Neither blocks nor log segments found. Creating new log.
I20260928 05:17:39.509584 5512 tablet_bootstrap.cc:492] T d94779c029c54c4fa26fd8077624f676 P 0d866a0915e643b689492d22b6a9a1ed: Bootstrap starting.
I20260928 05:17:39.510705 5511 log.cc:824] T d94779c029c54c4fa26fd8077624f676 P eca28ad6d2534d48b3e71750317a4dc3: Log is configured to *not* fsync() on all Append() calls
I20260928 05:17:39.512643 5514 tablet_bootstrap.cc:492] T d94779c029c54c4fa26fd8077624f676 P 04478c3d0d6a44bc8fc7c61b5947428b: Bootstrap starting.
I20260928 05:17:39.512990 5512 tablet_bootstrap.cc:654] T d94779c029c54c4fa26fd8077624f676 P 0d866a0915e643b689492d22b6a9a1ed: Neither blocks nor log segments found. Creating new log.
I20260928 05:17:39.513564 5511 tablet_bootstrap.cc:492] T d94779c029c54c4fa26fd8077624f676 P eca28ad6d2534d48b3e71750317a4dc3: No bootstrap required, opened a new log
I20260928 05:17:39.513808 5511 ts_tablet_manager.cc:1397] T d94779c029c54c4fa26fd8077624f676 P eca28ad6d2534d48b3e71750317a4dc3: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.005s
I20260928 05:17:39.514173 5512 log.cc:824] T d94779c029c54c4fa26fd8077624f676 P 0d866a0915e643b689492d22b6a9a1ed: Log is configured to *not* fsync() on all Append() calls
I20260928 05:17:39.515467 5514 tablet_bootstrap.cc:654] T d94779c029c54c4fa26fd8077624f676 P 04478c3d0d6a44bc8fc7c61b5947428b: Neither blocks nor log segments found. Creating new log.
I20260928 05:17:39.516757 5514 log.cc:824] T d94779c029c54c4fa26fd8077624f676 P 04478c3d0d6a44bc8fc7c61b5947428b: Log is configured to *not* fsync() on all Append() calls
I20260928 05:17:39.517192 5512 tablet_bootstrap.cc:492] T d94779c029c54c4fa26fd8077624f676 P 0d866a0915e643b689492d22b6a9a1ed: No bootstrap required, opened a new log
I20260928 05:17:39.517426 5512 ts_tablet_manager.cc:1397] T d94779c029c54c4fa26fd8077624f676 P 0d866a0915e643b689492d22b6a9a1ed: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.003s
I20260928 05:17:39.521973 5514 tablet_bootstrap.cc:492] T d94779c029c54c4fa26fd8077624f676 P 04478c3d0d6a44bc8fc7c61b5947428b: No bootstrap required, opened a new log
I20260928 05:17:39.522182 5514 ts_tablet_manager.cc:1397] T d94779c029c54c4fa26fd8077624f676 P 04478c3d0d6a44bc8fc7c61b5947428b: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260928 05:17:39.523254 5511 raft_consensus.cc:348] T d94779c029c54c4fa26fd8077624f676 P eca28ad6d2534d48b3e71750317a4dc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d866a0915e643b689492d22b6a9a1ed" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32947 } } peers { permanent_uuid: "04478c3d0d6a44bc8fc7c61b5947428b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44387 } } peers { permanent_uuid: "eca28ad6d2534d48b3e71750317a4dc3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38973 } }
I20260928 05:17:39.523654 5511 raft_consensus.cc:374] T d94779c029c54c4fa26fd8077624f676 P eca28ad6d2534d48b3e71750317a4dc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:17:39.523742 5511 raft_consensus.cc:729] T d94779c029c54c4fa26fd8077624f676 P eca28ad6d2534d48b3e71750317a4dc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eca28ad6d2534d48b3e71750317a4dc3, State: Initialized, Role: FOLLOWER
I20260928 05:17:39.524291 5511 consensus_queue.cc:260] T d94779c029c54c4fa26fd8077624f676 P eca28ad6d2534d48b3e71750317a4dc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d866a0915e643b689492d22b6a9a1ed" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32947 } } peers { permanent_uuid: "04478c3d0d6a44bc8fc7c61b5947428b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44387 } } peers { permanent_uuid: "eca28ad6d2534d48b3e71750317a4dc3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38973 } }
I20260928 05:17:39.525318 5233 heartbeater.cc:503] Master 127.0.97.62:43437 was elected leader, sending a full tablet report...
I20260928 05:17:39.525867 5511 ts_tablet_manager.cc:1428] T d94779c029c54c4fa26fd8077624f676 P eca28ad6d2534d48b3e71750317a4dc3: Time spent starting tablet: real 0.012s user 0.009s sys 0.003s
I20260928 05:17:39.526772 5512 raft_consensus.cc:348] T d94779c029c54c4fa26fd8077624f676 P 0d866a0915e643b689492d22b6a9a1ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d866a0915e643b689492d22b6a9a1ed" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32947 } } peers { permanent_uuid: "04478c3d0d6a44bc8fc7c61b5947428b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44387 } } peers { permanent_uuid: "eca28ad6d2534d48b3e71750317a4dc3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38973 } }
I20260928 05:17:39.527073 5512 raft_consensus.cc:374] T d94779c029c54c4fa26fd8077624f676 P 0d866a0915e643b689492d22b6a9a1ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:17:39.527166 5512 raft_consensus.cc:729] T d94779c029c54c4fa26fd8077624f676 P 0d866a0915e643b689492d22b6a9a1ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d866a0915e643b689492d22b6a9a1ed, State: Initialized, Role: FOLLOWER
I20260928 05:17:39.527649 5512 consensus_queue.cc:260] T d94779c029c54c4fa26fd8077624f676 P 0d866a0915e643b689492d22b6a9a1ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d866a0915e643b689492d22b6a9a1ed" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32947 } } peers { permanent_uuid: "04478c3d0d6a44bc8fc7c61b5947428b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44387 } } peers { permanent_uuid: "eca28ad6d2534d48b3e71750317a4dc3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38973 } }
I20260928 05:17:39.528532 5493 heartbeater.cc:503] Master 127.0.97.62:43437 was elected leader, sending a full tablet report...
I20260928 05:17:39.529120 5512 ts_tablet_manager.cc:1428] T d94779c029c54c4fa26fd8077624f676 P 0d866a0915e643b689492d22b6a9a1ed: Time spent starting tablet: real 0.012s user 0.011s sys 0.001s
I20260928 05:17:39.531329 5514 raft_consensus.cc:348] T d94779c029c54c4fa26fd8077624f676 P 04478c3d0d6a44bc8fc7c61b5947428b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d866a0915e643b689492d22b6a9a1ed" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32947 } } peers { permanent_uuid: "04478c3d0d6a44bc8fc7c61b5947428b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44387 } } peers { permanent_uuid: "eca28ad6d2534d48b3e71750317a4dc3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38973 } }
I20260928 05:17:39.531615 5514 raft_consensus.cc:374] T d94779c029c54c4fa26fd8077624f676 P 04478c3d0d6a44bc8fc7c61b5947428b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:17:39.531714 5514 raft_consensus.cc:729] T d94779c029c54c4fa26fd8077624f676 P 04478c3d0d6a44bc8fc7c61b5947428b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04478c3d0d6a44bc8fc7c61b5947428b, State: Initialized, Role: FOLLOWER
I20260928 05:17:39.532148 5514 consensus_queue.cc:260] T d94779c029c54c4fa26fd8077624f676 P 04478c3d0d6a44bc8fc7c61b5947428b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d866a0915e643b689492d22b6a9a1ed" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32947 } } peers { permanent_uuid: "04478c3d0d6a44bc8fc7c61b5947428b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44387 } } peers { permanent_uuid: "eca28ad6d2534d48b3e71750317a4dc3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38973 } }
I20260928 05:17:39.532471 5517 raft_consensus.cc:482] T d94779c029c54c4fa26fd8077624f676 P eca28ad6d2534d48b3e71750317a4dc3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:17:39.532773 5517 raft_consensus.cc:504] T d94779c029c54c4fa26fd8077624f676 P eca28ad6d2534d48b3e71750317a4dc3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d866a0915e643b689492d22b6a9a1ed" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32947 } } peers { permanent_uuid: "04478c3d0d6a44bc8fc7c61b5947428b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44387 } } peers { permanent_uuid: "eca28ad6d2534d48b3e71750317a4dc3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38973 } }
I20260928 05:17:39.533162 5363 heartbeater.cc:503] Master 127.0.97.62:43437 was elected leader, sending a full tablet report...
I20260928 05:17:39.533717 5514 ts_tablet_manager.cc:1428] T d94779c029c54c4fa26fd8077624f676 P 04478c3d0d6a44bc8fc7c61b5947428b: Time spent starting tablet: real 0.011s user 0.005s sys 0.006s
I20260928 05:17:39.534452 5517 leader_election.cc:288] T d94779c029c54c4fa26fd8077624f676 P eca28ad6d2534d48b3e71750317a4dc3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d866a0915e643b689492d22b6a9a1ed (127.0.97.3:32947), 04478c3d0d6a44bc8fc7c61b5947428b (127.0.97.2:44387)
I20260928 05:17:39.537580 5508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:39.535093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48519 (local address 127.0.97.2:44387)
0928 05:17:39.535233 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:39.535238 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:39.535333 (+ 95us) server_negotiation.cc:407] Connection header received
0928 05:17:39.535535 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:39.535541 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:39.535594 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:39.535659 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:39.537049 (+ 1390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:39.537055 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:39.537060 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:17:39.537076 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:39.537105 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:17:39.537115 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:39.537210 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:39.537295 (+ 85us) server_negotiation.cc:299] Negotiation successful
0928 05:17:39.537348 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 05:17:39.537787 5520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:39.534700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44387 (local address 127.0.97.1:48519)
0928 05:17:39.535248 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:39.535266 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:39.535408 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:39.535854 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:39.535859 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:39.535899 (+ 40us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:17:39.536822 (+ 923us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 05:17:39.536825 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:17:39.536841 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 05:17:39.536913 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:17:39.537197 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:17:39.537201 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:17:39.537206 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 05:17:39.537335 (+ 129us) client_negotiation.cc:240] Negotiation successful
0928 05:17:39.537378 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":541,"threads_started":1}
I20260928 05:17:39.538590 5318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d94779c029c54c4fa26fd8077624f676" candidate_uuid: "eca28ad6d2534d48b3e71750317a4dc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04478c3d0d6a44bc8fc7c61b5947428b" is_pre_election: true
I20260928 05:17:39.538820 5507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:39.535389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51699 (local address 127.0.97.3:32947)
0928 05:17:39.535545 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:39.535550 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:39.537117 (+ 1567us) server_negotiation.cc:407] Connection header received
0928 05:17:39.537241 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:39.537246 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:39.537290 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:39.537356 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:39.538280 (+ 924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:39.538286 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:39.538290 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:17:39.538308 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:39.538344 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:17:39.538359 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:39.538475 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:39.538548 (+ 73us) server_negotiation.cc:299] Negotiation successful
0928 05:17:39.538601 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 05:17:39.538973 5521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:39.534723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:32947 (local address 127.0.97.1:51699)
0928 05:17:39.537040 (+ 2317us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:39.537057 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:39.537150 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:39.537392 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:39.537397 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:39.537422 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:17:39.538094 (+ 672us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 05:17:39.538097 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:17:39.538112 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 05:17:39.538150 (+ 38us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:17:39.538454 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:17:39.538457 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:17:39.538460 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 05:17:39.538540 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 05:17:39.538576 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2199,"thread_start_us":1225,"threads_started":1}
I20260928 05:17:39.538897 5518 raft_consensus.cc:482] T d94779c029c54c4fa26fd8077624f676 P 0d866a0915e643b689492d22b6a9a1ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:17:39.539067 5318 raft_consensus.cc:2454] T d94779c029c54c4fa26fd8077624f676 P 04478c3d0d6a44bc8fc7c61b5947428b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eca28ad6d2534d48b3e71750317a4dc3 in term 0.
I20260928 05:17:39.539122 5518 raft_consensus.cc:504] T d94779c029c54c4fa26fd8077624f676 P 0d866a0915e643b689492d22b6a9a1ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d866a0915e643b689492d22b6a9a1ed" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32947 } } peers { permanent_uuid: "04478c3d0d6a44bc8fc7c61b5947428b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44387 } } peers { permanent_uuid: "eca28ad6d2534d48b3e71750317a4dc3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38973 } }
I20260928 05:17:39.539682 5123 leader_election.cc:302] T d94779c029c54c4fa26fd8077624f676 P eca28ad6d2534d48b3e71750317a4dc3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04478c3d0d6a44bc8fc7c61b5947428b, eca28ad6d2534d48b3e71750317a4dc3; no voters:
I20260928 05:17:39.539815 5448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d94779c029c54c4fa26fd8077624f676" candidate_uuid: "eca28ad6d2534d48b3e71750317a4dc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d866a0915e643b689492d22b6a9a1ed" is_pre_election: true
I20260928 05:17:39.540040 5517 raft_consensus.cc:2781] T d94779c029c54c4fa26fd8077624f676 P eca28ad6d2534d48b3e71750317a4dc3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 05:17:39.540160 5517 raft_consensus.cc:482] T d94779c029c54c4fa26fd8077624f676 P eca28ad6d2534d48b3e71750317a4dc3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 05:17:39.540249 5517 raft_consensus.cc:3037] T d94779c029c54c4fa26fd8077624f676 P eca28ad6d2534d48b3e71750317a4dc3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:17:39.540249 5448 raft_consensus.cc:2454] T d94779c029c54c4fa26fd8077624f676 P 0d866a0915e643b689492d22b6a9a1ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eca28ad6d2534d48b3e71750317a4dc3 in term 0.
W20260928 05:17:39.540764 5364 tablet.cc:2369] T d94779c029c54c4fa26fd8077624f676 P 04478c3d0d6a44bc8fc7c61b5947428b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 05:17:39.542928 5508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:39.541042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58795 (local address 127.0.97.2:44387)
0928 05:17:39.541183 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:39.541189 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:39.541203 (+ 14us) server_negotiation.cc:407] Connection header received
0928 05:17:39.541242 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:39.541246 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:39.541309 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:39.541392 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:39.542345 (+ 953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:39.542350 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:39.542355 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:17:39.542371 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:39.542403 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:17:39.542419 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:39.542544 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:39.542620 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 05:17:39.542666 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 05:17:39.542979 5495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:39.540743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44387 (local address 127.0.97.3:58795)
0928 05:17:39.541041 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:39.541054 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:39.541120 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:39.541569 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:39.541573 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:39.541595 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:17:39.542139 (+ 544us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 05:17:39.542143 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:17:39.542161 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 05:17:39.542207 (+ 46us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:17:39.542511 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:17:39.542515 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:17:39.542520 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 05:17:39.542625 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 05:17:39.542675 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200}
I20260928 05:17:39.543375 5517 raft_consensus.cc:504] T d94779c029c54c4fa26fd8077624f676 P eca28ad6d2534d48b3e71750317a4dc3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d866a0915e643b689492d22b6a9a1ed" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32947 } } peers { permanent_uuid: "04478c3d0d6a44bc8fc7c61b5947428b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44387 } } peers { permanent_uuid: "eca28ad6d2534d48b3e71750317a4dc3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38973 } }
I20260928 05:17:39.543898 5318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d94779c029c54c4fa26fd8077624f676" candidate_uuid: "0d866a0915e643b689492d22b6a9a1ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04478c3d0d6a44bc8fc7c61b5947428b" is_pre_election: true
I20260928 05:17:39.544094 5318 raft_consensus.cc:2454] T d94779c029c54c4fa26fd8077624f676 P 04478c3d0d6a44bc8fc7c61b5947428b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d866a0915e643b689492d22b6a9a1ed in term 0.
I20260928 05:17:39.544373 5317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d94779c029c54c4fa26fd8077624f676" candidate_uuid: "eca28ad6d2534d48b3e71750317a4dc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04478c3d0d6a44bc8fc7c61b5947428b"
I20260928 05:17:39.544572 5383 leader_election.cc:302] T d94779c029c54c4fa26fd8077624f676 P 0d866a0915e643b689492d22b6a9a1ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04478c3d0d6a44bc8fc7c61b5947428b, 0d866a0915e643b689492d22b6a9a1ed; no voters:
I20260928 05:17:39.544601 5317 raft_consensus.cc:3037] T d94779c029c54c4fa26fd8077624f676 P 04478c3d0d6a44bc8fc7c61b5947428b [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:17:39.544613 5448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d94779c029c54c4fa26fd8077624f676" candidate_uuid: "eca28ad6d2534d48b3e71750317a4dc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d866a0915e643b689492d22b6a9a1ed"
I20260928 05:17:39.544828 5448 raft_consensus.cc:3037] T d94779c029c54c4fa26fd8077624f676 P 0d866a0915e643b689492d22b6a9a1ed [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:17:39.545217 5510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:39.541515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46063 (local address 127.0.97.1:38973)
0928 05:17:39.541668 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:39.541673 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:39.543244 (+ 1571us) server_negotiation.cc:407] Connection header received
0928 05:17:39.543353 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:39.543358 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:39.543420 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:39.543503 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:39.544327 (+ 824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:39.544332 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:39.544337 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:17:39.544353 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:39.544392 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:17:39.544406 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:39.544498 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:39.544919 (+ 421us) server_negotiation.cc:299] Negotiation successful
0928 05:17:39.544968 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 05:17:39.547565 5317 raft_consensus.cc:2454] T d94779c029c54c4fa26fd8077624f676 P 04478c3d0d6a44bc8fc7c61b5947428b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eca28ad6d2534d48b3e71750317a4dc3 in term 1.
I20260928 05:17:39.548105 5123 leader_election.cc:302] T d94779c029c54c4fa26fd8077624f676 P eca28ad6d2534d48b3e71750317a4dc3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04478c3d0d6a44bc8fc7c61b5947428b, eca28ad6d2534d48b3e71750317a4dc3; no voters:
I20260928 05:17:39.549151 5495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:39.541218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38973 (local address 127.0.97.3:46063)
0928 05:17:39.543176 (+ 1958us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:39.543193 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:39.543268 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:39.543640 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:39.543643 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:39.543659 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:17:39.544130 (+ 471us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 05:17:39.544134 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:17:39.544149 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 05:17:39.544187 (+ 38us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:17:39.544543 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:17:39.544548 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:17:39.544553 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 05:17:39.544668 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 05:17:39.544717 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1860,"threads_started":1}
I20260928 05:17:39.549213 5518 leader_election.cc:288] T d94779c029c54c4fa26fd8077624f676 P 0d866a0915e643b689492d22b6a9a1ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 04478c3d0d6a44bc8fc7c61b5947428b (127.0.97.2:44387), eca28ad6d2534d48b3e71750317a4dc3 (127.0.97.1:38973)
I20260928 05:17:39.549276 5448 raft_consensus.cc:2454] T d94779c029c54c4fa26fd8077624f676 P 0d866a0915e643b689492d22b6a9a1ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eca28ad6d2534d48b3e71750317a4dc3 in term 1.
I20260928 05:17:39.549739 5518 raft_consensus.cc:2741] T d94779c029c54c4fa26fd8077624f676 P 0d866a0915e643b689492d22b6a9a1ed [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260928 05:17:39.549851 5517 leader_election.cc:288] T d94779c029c54c4fa26fd8077624f676 P eca28ad6d2534d48b3e71750317a4dc3 [CANDIDATE]: Term 1 election: Requested vote from peers 0d866a0915e643b689492d22b6a9a1ed (127.0.97.3:32947), 04478c3d0d6a44bc8fc7c61b5947428b (127.0.97.2:44387)
I20260928 05:17:39.550137 5517 raft_consensus.cc:2781] T d94779c029c54c4fa26fd8077624f676 P eca28ad6d2534d48b3e71750317a4dc3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:17:39.550246 5517 raft_consensus.cc:686] T d94779c029c54c4fa26fd8077624f676 P eca28ad6d2534d48b3e71750317a4dc3 [term 1 LEADER]: Becoming Leader. State: Replica: eca28ad6d2534d48b3e71750317a4dc3, State: Running, Role: LEADER
I20260928 05:17:39.550611 5517 consensus_queue.cc:237] T d94779c029c54c4fa26fd8077624f676 P eca28ad6d2534d48b3e71750317a4dc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d866a0915e643b689492d22b6a9a1ed" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32947 } } peers { permanent_uuid: "04478c3d0d6a44bc8fc7c61b5947428b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44387 } } peers { permanent_uuid: "eca28ad6d2534d48b3e71750317a4dc3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38973 } }
I20260928 05:17:39.554653 5188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d94779c029c54c4fa26fd8077624f676" candidate_uuid: "0d866a0915e643b689492d22b6a9a1ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eca28ad6d2534d48b3e71750317a4dc3" is_pre_election: true
I20260928 05:17:39.556105 5046 catalog_manager.cc:5495] T d94779c029c54c4fa26fd8077624f676 P eca28ad6d2534d48b3e71750317a4dc3 reported cstate change: term changed from 0 to 1, leader changed from <none> to eca28ad6d2534d48b3e71750317a4dc3 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "eca28ad6d2534d48b3e71750317a4dc3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d866a0915e643b689492d22b6a9a1ed" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "04478c3d0d6a44bc8fc7c61b5947428b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eca28ad6d2534d48b3e71750317a4dc3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38973 } health_report { overall_health: HEALTHY } } }
I20260928 05:17:39.571522 5510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:39.569602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60456 (local address 127.0.97.1:38973)
0928 05:17:39.569776 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:39.569782 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:39.569883 (+ 101us) server_negotiation.cc:407] Connection header received
0928 05:17:39.570100 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:39.570103 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:39.570183 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:39.570265 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:39.570979 (+ 714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:39.570984 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:39.570989 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:17:39.571006 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:39.571036 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:17:39.571047 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:39.571143 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:39.571239 (+ 96us) server_negotiation.cc:299] Negotiation successful
0928 05:17:39.571288 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 05:17:39.578409 5448 raft_consensus.cc:1261] T d94779c029c54c4fa26fd8077624f676 P 0d866a0915e643b689492d22b6a9a1ed [term 1 FOLLOWER]: Refusing update from remote peer eca28ad6d2534d48b3e71750317a4dc3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:17:39.578540 5317 raft_consensus.cc:1261] T d94779c029c54c4fa26fd8077624f676 P 04478c3d0d6a44bc8fc7c61b5947428b [term 1 FOLLOWER]: Refusing update from remote peer eca28ad6d2534d48b3e71750317a4dc3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:17:39.579150 5524 consensus_queue.cc:1035] T d94779c029c54c4fa26fd8077624f676 P eca28ad6d2534d48b3e71750317a4dc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d866a0915e643b689492d22b6a9a1ed" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:17:39.579457 5517 consensus_queue.cc:1035] T d94779c029c54c4fa26fd8077624f676 P eca28ad6d2534d48b3e71750317a4dc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04478c3d0d6a44bc8fc7c61b5947428b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:17:39.589089 5529 mvcc.cc:205] Tried to move back new op lower bound from 7334185613625069568 to 7334185613522333696. Current Snapshot: MvccSnapshot[applied={T|T < 7334185613625069568}]
I20260928 05:17:39.592419 5528 mvcc.cc:205] Tried to move back new op lower bound from 7334185613625069568 to 7334185613522333696. Current Snapshot: MvccSnapshot[applied={T|T < 7334185613625069568}]
I20260928 05:17:39.600975 5047 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47344:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:17:39.601199 5047 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47344:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:17:39.604317 5047 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 64be312980ad41e1949184b33c4537c6: Sending DeleteTablet for 3 replicas of tablet d94779c029c54c4fa26fd8077624f676
I20260928 05:17:39.605352 5428 tablet_service.cc:1514] Processing DeleteTablet for tablet d94779c029c54c4fa26fd8077624f676 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:17:39 UTC) from {username='slave'} at 127.0.0.1:37426
I20260928 05:17:39.605376 5298 tablet_service.cc:1514] Processing DeleteTablet for tablet d94779c029c54c4fa26fd8077624f676 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:17:39 UTC) from {username='slave'} at 127.0.0.1:46808
I20260928 05:17:39.606411 5168 tablet_service.cc:1514] Processing DeleteTablet for tablet d94779c029c54c4fa26fd8077624f676 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:17:39 UTC) from {username='slave'} at 127.0.0.1:60452
I20260928 05:17:39.609213 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 5106
I20260928 05:17:39.619202 5538 tablet_replica.cc:330] T d94779c029c54c4fa26fd8077624f676 P eca28ad6d2534d48b3e71750317a4dc3: stopping tablet replica
I20260928 05:17:39.619688 5536 tablet_replica.cc:330] T d94779c029c54c4fa26fd8077624f676 P 0d866a0915e643b689492d22b6a9a1ed: stopping tablet replica
I20260928 05:17:39.620224 5536 raft_consensus.cc:2229] T d94779c029c54c4fa26fd8077624f676 P 0d866a0915e643b689492d22b6a9a1ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:17:39.620671 5538 raft_consensus.cc:2229] T d94779c029c54c4fa26fd8077624f676 P eca28ad6d2534d48b3e71750317a4dc3 [term 1 LEADER]: Raft consensus shutting down.
I20260928 05:17:39.621275 5536 raft_consensus.cc:2258] T d94779c029c54c4fa26fd8077624f676 P 0d866a0915e643b689492d22b6a9a1ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:17:39.621908 5538 raft_consensus.cc:2258] T d94779c029c54c4fa26fd8077624f676 P eca28ad6d2534d48b3e71750317a4dc3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:17:39.623986 5536 ts_tablet_manager.cc:1905] T d94779c029c54c4fa26fd8077624f676 P 0d866a0915e643b689492d22b6a9a1ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:17:39.625749 5538 ts_tablet_manager.cc:1905] T d94779c029c54c4fa26fd8077624f676 P eca28ad6d2534d48b3e71750317a4dc3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:17:39.631376 5536 ts_tablet_manager.cc:1918] T d94779c029c54c4fa26fd8077624f676 P 0d866a0915e643b689492d22b6a9a1ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:17:39.631588 5536 log.cc:1196] T d94779c029c54c4fa26fd8077624f676 P 0d866a0915e643b689492d22b6a9a1ed: Deleting WAL directory at /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572622576160-388-0/minicluster-data/ts-2/wal/wals/d94779c029c54c4fa26fd8077624f676
I20260928 05:17:39.632157 5536 ts_tablet_manager.cc:1939] T d94779c029c54c4fa26fd8077624f676 P 0d866a0915e643b689492d22b6a9a1ed: Deleting consensus metadata
I20260928 05:17:39.632773 5538 ts_tablet_manager.cc:1918] T d94779c029c54c4fa26fd8077624f676 P eca28ad6d2534d48b3e71750317a4dc3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:17:39.633037 5538 log.cc:1196] T d94779c029c54c4fa26fd8077624f676 P eca28ad6d2534d48b3e71750317a4dc3: Deleting WAL directory at /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572622576160-388-0/minicluster-data/ts-0/wal/wals/d94779c029c54c4fa26fd8077624f676
I20260928 05:17:39.633554 5538 ts_tablet_manager.cc:1939] T d94779c029c54c4fa26fd8077624f676 P eca28ad6d2534d48b3e71750317a4dc3: Deleting consensus metadata
I20260928 05:17:39.633988 5032 catalog_manager.cc:4841] TS 0d866a0915e643b689492d22b6a9a1ed (127.0.97.3:32947): tablet d94779c029c54c4fa26fd8077624f676 (table test-table [id=6b4aac3d369d4785ae82b701b2174aaa]) successfully deleted
I20260928 05:17:39.634696 5537 tablet_replica.cc:330] T d94779c029c54c4fa26fd8077624f676 P 04478c3d0d6a44bc8fc7c61b5947428b: stopping tablet replica
I20260928 05:17:39.635138 5537 raft_consensus.cc:2229] T d94779c029c54c4fa26fd8077624f676 P 04478c3d0d6a44bc8fc7c61b5947428b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:17:39.635586 5537 raft_consensus.cc:2258] T d94779c029c54c4fa26fd8077624f676 P 04478c3d0d6a44bc8fc7c61b5947428b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:17:39.635574 5031 catalog_manager.cc:4841] TS eca28ad6d2534d48b3e71750317a4dc3 (127.0.97.1:38973): tablet d94779c029c54c4fa26fd8077624f676 (table test-table [id=6b4aac3d369d4785ae82b701b2174aaa]) successfully deleted
I20260928 05:17:39.636168 5510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:39.612137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60464 (local address 127.0.97.1:38973)
0928 05:17:39.612310 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:39.612317 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:39.633614 (+ 21297us) server_negotiation.cc:407] Connection header received
0928 05:17:39.634079 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:39.634084 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:39.634140 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:39.634209 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:39.635368 (+ 1159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:39.635373 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:39.635377 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:17:39.635394 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:39.635428 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:17:39.635443 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:39.635548 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:39.635860 (+ 312us) server_negotiation.cc:299] Negotiation successful
0928 05:17:39.635916 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 05:17:39.637269 5228 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:17:39.637928 5537 ts_tablet_manager.cc:1905] T d94779c029c54c4fa26fd8077624f676 P 04478c3d0d6a44bc8fc7c61b5947428b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:17:39.644591 5537 ts_tablet_manager.cc:1918] T d94779c029c54c4fa26fd8077624f676 P 04478c3d0d6a44bc8fc7c61b5947428b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:17:39.644786 5537 log.cc:1196] T d94779c029c54c4fa26fd8077624f676 P 04478c3d0d6a44bc8fc7c61b5947428b: Deleting WAL directory at /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572622576160-388-0/minicluster-data/ts-1/wal/wals/d94779c029c54c4fa26fd8077624f676
I20260928 05:17:39.645442 5537 ts_tablet_manager.cc:1939] T d94779c029c54c4fa26fd8077624f676 P 04478c3d0d6a44bc8fc7c61b5947428b: Deleting consensus metadata
I20260928 05:17:39.647156 5033 catalog_manager.cc:4841] TS 04478c3d0d6a44bc8fc7c61b5947428b (127.0.97.2:44387): tablet d94779c029c54c4fa26fd8077624f676 (table test-table [id=6b4aac3d369d4785ae82b701b2174aaa]) successfully deleted
I20260928 05:17:39.743160 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 5106
I20260928 05:17:39.757038 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 5236
I20260928 05:17:39.759405 5508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:39.757727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46810 (local address 127.0.97.2:44387)
0928 05:17:39.757891 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:39.757895 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:39.757912 (+ 17us) server_negotiation.cc:407] Connection header received
0928 05:17:39.758044 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:39.758048 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:39.758113 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:39.758190 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:39.758912 (+ 722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:39.758915 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:39.758918 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:17:39.758929 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:39.758976 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:17:39.758986 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:39.759064 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:39.759138 (+ 74us) server_negotiation.cc:299] Negotiation successful
0928 05:17:39.759177 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 05:17:39.759980 5358 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:17:39.842224 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 5236
I20260928 05:17:39.854604 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 5366
I20260928 05:17:39.857034 5507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:39.855270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37436 (local address 127.0.97.3:32947)
0928 05:17:39.855451 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:39.855456 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:39.855473 (+ 17us) server_negotiation.cc:407] Connection header received
0928 05:17:39.855617 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:39.855621 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:39.855704 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:39.855779 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:39.856492 (+ 713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:39.856496 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:39.856499 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:17:39.856514 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:39.856543 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:17:39.856555 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:39.856650 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:39.856687 (+ 37us) server_negotiation.cc:299] Negotiation successful
0928 05:17:39.856730 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 05:17:39.857650 5488 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:17:39.941651 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 5366
I20260928 05:17:39.954370 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 5016
I20260928 05:17:39.955423 5077 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:17:40.054148 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 5016
2026-09-28T05:17:40Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2313 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260928 05:17:40.088939 388 mini_oidc.cc:152] Starting JWKS server
I20260928 05:17:40.091275 388 webserver.cc:460] Webserver started at https://127.0.0.1:43377/ using document root <none> and password file <none>
I20260928 05:17:40.091530 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 05:17:40.092062 388 webserver.cc:460] Webserver started at http://127.0.0.1:43587/ using document root <none> and password file <none>
2026-09-28T05:17:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:17:40Z Disabled control of system clock
I20260928 05:17:40.113232 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790572622576160-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790572622576160-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790572622576160-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44781
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:46405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44781
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43377/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790572622576160-388-0/testchainca.cert with env {}
W20260928 05:17:40.404884 5549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:17:40.405135 5549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:17:40.405225 5549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:17:40.412056 5549 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:17:40.412143 5549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:17:40.412186 5549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:17:40.412236 5549 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:17:40.412277 5549 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 05:17:40.412333 5549 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 05:17:40.412370 5549 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43377/jwks.json
I20260928 05:17:40.421406 5549 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790572622576160-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790572622576160-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44781
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43377/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790572622576160-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790572622576160-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:17:40.422462 5549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:17:40.423576 5549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:17:40.434070 5554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:40.434165 5555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:40.434140 5557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:17:40.434968 5549 server_base.cc:1034] running on GCE node
I20260928 05:17:40.435578 5549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:17:40.436770 5549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:17:40.437965 5549 hybrid_clock.cc:648] HybridClock initialized: now 1790572660437928 us; error 29 us; skew 500 ppm
I20260928 05:17:40.438248 5549 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 05:17:40.440217 5549 webserver.cc:460] Webserver started at http://127.0.97.62:37311/ using document root <none> and password file <none>
I20260928 05:17:40.440776 5549 fs_manager.cc:359] Metadata directory not provided
I20260928 05:17:40.440888 5549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:17:40.441139 5549 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:17:40.443661 5549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790572622576160-388-0/minicluster-data/master-0/data/instance:
uuid: "f10a1f0ddc3342918f3d12b3a3515564"
format_stamp: "Formatted at 2026-09-28 05:17:40 on dist-test-slave-whsn"
server_key: "a2a360de812d7be7d595a990d57ad454"
server_key_iv: "759d54d73fb2498646381eff3c4b5bd0"
server_key_version: "encryptionkey@0"
I20260928 05:17:40.444352 5549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790572622576160-388-0/minicluster-data/master-0/wal/instance:
uuid: "f10a1f0ddc3342918f3d12b3a3515564"
format_stamp: "Formatted at 2026-09-28 05:17:40 on dist-test-slave-whsn"
server_key: "a2a360de812d7be7d595a990d57ad454"
server_key_iv: "759d54d73fb2498646381eff3c4b5bd0"
server_key_version: "encryptionkey@0"
I20260928 05:17:40.448776 5549 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.007s
I20260928 05:17:40.451694 5563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:40.452929 5549 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 05:17:40.453085 5549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790572622576160-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790572622576160-388-0/minicluster-data/master-0/wal
uuid: "f10a1f0ddc3342918f3d12b3a3515564"
format_stamp: "Formatted at 2026-09-28 05:17:40 on dist-test-slave-whsn"
server_key: "a2a360de812d7be7d595a990d57ad454"
server_key_iv: "759d54d73fb2498646381eff3c4b5bd0"
server_key_version: "encryptionkey@0"
I20260928 05:17:40.453259 5549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790572622576160-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790572622576160-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790572622576160-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:17:40.466276 5549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:17:40.466969 5549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:17:40.467190 5549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:17:40.482472 5549 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44781
I20260928 05:17:40.482499 5614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44781 every 8 connection(s)
I20260928 05:17:40.483568 5549 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:17:40.486840 5615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:17:40.487253 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 5549
I20260928 05:17:40.487416 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790572622576160-388-0/minicluster-data/master-0/wal/instance
I20260928 05:17:40.487990 388 external_mini_cluster.cc:1428] Setting key 88894af4ab0751cdffbf83baff50fe7e
I20260928 05:17:40.500341 5615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f10a1f0ddc3342918f3d12b3a3515564: Bootstrap starting.
I20260928 05:17:40.503436 5618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:40.489607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39718 (local address 127.0.97.62:44781)
0928 05:17:40.490559 (+ 952us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:40.490576 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:40.490622 (+ 46us) server_negotiation.cc:407] Connection header received
0928 05:17:40.492756 (+ 2134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:40.492781 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:40.493366 (+ 585us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:40.493791 (+ 425us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:40.494856 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:40.496805 (+ 1949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:40.498702 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:40.499647 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:40.499804 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:40.500977 (+ 1173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:40.500987 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:40.501003 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:17:40.501630 (+ 627us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:40.501732 (+ 102us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:17:40.501758 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:40.501884 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:40.502033 (+ 149us) server_negotiation.cc:299] Negotiation successful
0928 05:17:40.502329 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":250,"threads_started":1}
I20260928 05:17:40.503903 5615 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f10a1f0ddc3342918f3d12b3a3515564: Neither blocks nor log segments found. Creating new log.
I20260928 05:17:40.504846 5615 log.cc:824] T 00000000000000000000000000000000 P f10a1f0ddc3342918f3d12b3a3515564: Log is configured to *not* fsync() on all Append() calls
I20260928 05:17:40.506808 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 05:17:40.507372 5615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f10a1f0ddc3342918f3d12b3a3515564: No bootstrap required, opened a new log
I20260928 05:17:40.514288 5615 raft_consensus.cc:348] T 00000000000000000000000000000000 P f10a1f0ddc3342918f3d12b3a3515564 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f10a1f0ddc3342918f3d12b3a3515564" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44781 } }
I20260928 05:17:40.514575 5615 raft_consensus.cc:374] T 00000000000000000000000000000000 P f10a1f0ddc3342918f3d12b3a3515564 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:17:40.514664 5615 raft_consensus.cc:729] T 00000000000000000000000000000000 P f10a1f0ddc3342918f3d12b3a3515564 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f10a1f0ddc3342918f3d12b3a3515564, State: Initialized, Role: FOLLOWER
I20260928 05:17:40.515165 5615 consensus_queue.cc:260] T 00000000000000000000000000000000 P f10a1f0ddc3342918f3d12b3a3515564 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f10a1f0ddc3342918f3d12b3a3515564" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44781 } }
I20260928 05:17:40.515357 5615 raft_consensus.cc:388] T 00000000000000000000000000000000 P f10a1f0ddc3342918f3d12b3a3515564 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:17:40.515471 5615 raft_consensus.cc:482] T 00000000000000000000000000000000 P f10a1f0ddc3342918f3d12b3a3515564 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:17:40.515615 5615 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f10a1f0ddc3342918f3d12b3a3515564 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:17:40.517536 5615 raft_consensus.cc:504] T 00000000000000000000000000000000 P f10a1f0ddc3342918f3d12b3a3515564 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f10a1f0ddc3342918f3d12b3a3515564" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44781 } }
I20260928 05:17:40.517953 5615 leader_election.cc:302] T 00000000000000000000000000000000 P f10a1f0ddc3342918f3d12b3a3515564 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f10a1f0ddc3342918f3d12b3a3515564; no voters:
I20260928 05:17:40.518407 5615 leader_election.cc:288] T 00000000000000000000000000000000 P f10a1f0ddc3342918f3d12b3a3515564 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:17:40.518532 5620 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f10a1f0ddc3342918f3d12b3a3515564 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:17:40.519126 5620 raft_consensus.cc:686] T 00000000000000000000000000000000 P f10a1f0ddc3342918f3d12b3a3515564 [term 1 LEADER]: Becoming Leader. State: Replica: f10a1f0ddc3342918f3d12b3a3515564, State: Running, Role: LEADER
I20260928 05:17:40.519632 5620 consensus_queue.cc:237] T 00000000000000000000000000000000 P f10a1f0ddc3342918f3d12b3a3515564 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f10a1f0ddc3342918f3d12b3a3515564" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44781 } }
I20260928 05:17:40.520292 5615 sys_catalog.cc:564] T 00000000000000000000000000000000 P f10a1f0ddc3342918f3d12b3a3515564 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:17:40.522176 5621 sys_catalog.cc:455] T 00000000000000000000000000000000 P f10a1f0ddc3342918f3d12b3a3515564 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f10a1f0ddc3342918f3d12b3a3515564" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f10a1f0ddc3342918f3d12b3a3515564" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44781 } } }
I20260928 05:17:40.522519 5621 sys_catalog.cc:458] T 00000000000000000000000000000000 P f10a1f0ddc3342918f3d12b3a3515564 [sys.catalog]: This master's current role is: LEADER
I20260928 05:17:40.522780 5622 sys_catalog.cc:455] T 00000000000000000000000000000000 P f10a1f0ddc3342918f3d12b3a3515564 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f10a1f0ddc3342918f3d12b3a3515564. Latest consensus state: current_term: 1 leader_uuid: "f10a1f0ddc3342918f3d12b3a3515564" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f10a1f0ddc3342918f3d12b3a3515564" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44781 } } }
I20260928 05:17:40.523103 5622 sys_catalog.cc:458] T 00000000000000000000000000000000 P f10a1f0ddc3342918f3d12b3a3515564 [sys.catalog]: This master's current role is: LEADER
I20260928 05:17:40.524523 5627 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:17:40.529086 5627 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:17:40.536957 5627 catalog_manager.cc:1347] Generated new cluster ID: 16e0ad31a9ab47e498a4c93facc62806
I20260928 05:17:40.537040 5627 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:17:40.550029 5627 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:17:40.551141 5627 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:17:40.561488 5627 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f10a1f0ddc3342918f3d12b3a3515564: Generated new TSK 0
I20260928 05:17:40.562209 5627 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 05:17:40.603856 5645 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260928 05:17:40.603983 5645 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 05:17:40.605127 5618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:40.584638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39724 (local address 127.0.97.62:44781)
0928 05:17:40.584803 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:40.584810 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:40.585206 (+ 396us) server_negotiation.cc:407] Connection header received
0928 05:17:40.585471 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:40.585477 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:40.585562 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:40.585697 (+ 135us) server_negotiation.cc:226] Negotiated authn=JWT
0928 05:17:40.586139 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:40.587213 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:40.588511 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:40.589210 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:40.589345 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:40.589790 (+ 445us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 05:17:40.604470 (+ 14680us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43377/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 05:17:40.604808 (+ 338us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39724: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43377/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":51}
W20260928 05:17:40.605237 5618 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39724: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43377/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260928 05:17:40.606295 5644 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:17:40.584508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44781 (local address 127.0.0.1:39724)
0928 05:17:40.585120 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:40.585138 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:40.585271 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:40.585709 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:40.585712 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:40.585721 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0928 05:17:40.586016 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:40.586023 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:40.587315 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:40.587319 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:40.588388 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:40.588398 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:40.589327 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:40.589379 (+ 52us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:40.589505 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:40.589558 (+ 53us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 05:17:40.604703 (+ 15145us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43377/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 05:17:40.604967 (+ 264us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:44781: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43377/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":333,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.97.62:44781: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43377/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260928 05:17:40.607627 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 5549
I20260928 05:17:40.608657 5610 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:17:40.697635 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 5549
2026-09-28T05:17:40Z chronyd exiting
I20260928 05:17:40.743547 388 test_util.cc:175] -----------------------------------------------
I20260928 05:17:40.743656 388 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790572622576160-388-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (658 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260928 05:17:40.747115 388 mini_oidc.cc:152] Starting JWKS server
I20260928 05:17:40.748725 388 webserver.cc:460] Webserver started at https://127.0.0.1:37179/ using document root <none> and password file <none>
I20260928 05:17:40.748937 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 05:17:40.749500 388 webserver.cc:460] Webserver started at http://127.0.0.1:37023/ using document root <none> and password file <none>
2026-09-28T05:17:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:17:40Z Disabled control of system clock
I20260928 05:17:40.770706 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790572622576160-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790572622576160-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790572622576160-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39425
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:33493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39425
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37179/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790572622576160-388-0/testchainca.cert with env {}
W20260928 05:17:41.065236 5652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:17:41.065523 5652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:17:41.065636 5652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:17:41.072571 5652 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:17:41.072687 5652 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:17:41.072748 5652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:17:41.072829 5652 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:17:41.072903 5652 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 05:17:41.072986 5652 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 05:17:41.073050 5652 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37179/jwks.json
I20260928 05:17:41.082597 5652 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790572622576160-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790572622576160-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39425
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39425
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37179/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790572622576160-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790572622576160-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:17:41.083855 5652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:17:41.085137 5652 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:17:41.096256 5658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:41.096235 5657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:41.096225 5660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:17:41.097374 5652 server_base.cc:1034] running on GCE node
I20260928 05:17:41.098074 5652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:17:41.099373 5652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:17:41.100561 5652 hybrid_clock.cc:648] HybridClock initialized: now 1790572661100518 us; error 36 us; skew 500 ppm
I20260928 05:17:41.100903 5652 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 05:17:41.102941 5652 webserver.cc:460] Webserver started at http://127.0.97.62:41455/ using document root <none> and password file <none>
I20260928 05:17:41.103572 5652 fs_manager.cc:359] Metadata directory not provided
I20260928 05:17:41.103669 5652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:17:41.103955 5652 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:17:41.106680 5652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790572622576160-388-0/minicluster-data/master-0/data/instance:
uuid: "fbdcf043ba074ba894d1d04b323f5f08"
format_stamp: "Formatted at 2026-09-28 05:17:41 on dist-test-slave-whsn"
server_key: "6b386ef4da82daf904c21bdce1288292"
server_key_iv: "2ed75011f3c93563aff6bb5cc5dbef0f"
server_key_version: "encryptionkey@0"
I20260928 05:17:41.107419 5652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790572622576160-388-0/minicluster-data/master-0/wal/instance:
uuid: "fbdcf043ba074ba894d1d04b323f5f08"
format_stamp: "Formatted at 2026-09-28 05:17:41 on dist-test-slave-whsn"
server_key: "6b386ef4da82daf904c21bdce1288292"
server_key_iv: "2ed75011f3c93563aff6bb5cc5dbef0f"
server_key_version: "encryptionkey@0"
I20260928 05:17:41.112109 5652 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260928 05:17:41.115327 5666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:41.116597 5652 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 05:17:41.116752 5652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790572622576160-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790572622576160-388-0/minicluster-data/master-0/wal
uuid: "fbdcf043ba074ba894d1d04b323f5f08"
format_stamp: "Formatted at 2026-09-28 05:17:41 on dist-test-slave-whsn"
server_key: "6b386ef4da82daf904c21bdce1288292"
server_key_iv: "2ed75011f3c93563aff6bb5cc5dbef0f"
server_key_version: "encryptionkey@0"
I20260928 05:17:41.116979 5652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790572622576160-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790572622576160-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790572622576160-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:17:41.134181 5652 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:17:41.134934 5652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:17:41.135185 5652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:17:41.151273 5652 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39425
I20260928 05:17:41.151292 5717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39425 every 8 connection(s)
I20260928 05:17:41.152449 5652 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:17:41.153996 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 5652
I20260928 05:17:41.154172 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790572622576160-388-0/minicluster-data/master-0/wal/instance
I20260928 05:17:41.154745 388 external_mini_cluster.cc:1428] Setting key 411244def0a8f0d32ee831f6cb02a8b8
I20260928 05:17:41.156140 5718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:17:41.167193 5721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:41.156385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46044 (local address 127.0.97.62:39425)
0928 05:17:41.157296 (+ 911us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:41.157306 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:41.157344 (+ 38us) server_negotiation.cc:407] Connection header received
0928 05:17:41.158773 (+ 1429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:41.158792 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:41.159221 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:41.159542 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:41.160079 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:41.161398 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:41.163116 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:41.163736 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:41.163862 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:41.165058 (+ 1196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:41.165069 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:41.165092 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:17:41.165644 (+ 552us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:41.165723 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:17:41.165743 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:41.165830 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:41.166044 (+ 214us) server_negotiation.cc:299] Negotiation successful
0928 05:17:41.166331 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":335,"threads_started":1}
I20260928 05:17:41.168048 5718 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbdcf043ba074ba894d1d04b323f5f08: Bootstrap starting.
I20260928 05:17:41.170796 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 05:17:41.171483 5718 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fbdcf043ba074ba894d1d04b323f5f08: Neither blocks nor log segments found. Creating new log.
I20260928 05:17:41.172585 5718 log.cc:824] T 00000000000000000000000000000000 P fbdcf043ba074ba894d1d04b323f5f08: Log is configured to *not* fsync() on all Append() calls
I20260928 05:17:41.175233 5718 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbdcf043ba074ba894d1d04b323f5f08: No bootstrap required, opened a new log
I20260928 05:17:41.181892 5718 raft_consensus.cc:348] T 00000000000000000000000000000000 P fbdcf043ba074ba894d1d04b323f5f08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbdcf043ba074ba894d1d04b323f5f08" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39425 } }
I20260928 05:17:41.182289 5718 raft_consensus.cc:374] T 00000000000000000000000000000000 P fbdcf043ba074ba894d1d04b323f5f08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:17:41.182394 5718 raft_consensus.cc:729] T 00000000000000000000000000000000 P fbdcf043ba074ba894d1d04b323f5f08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbdcf043ba074ba894d1d04b323f5f08, State: Initialized, Role: FOLLOWER
I20260928 05:17:41.182893 5718 consensus_queue.cc:260] T 00000000000000000000000000000000 P fbdcf043ba074ba894d1d04b323f5f08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbdcf043ba074ba894d1d04b323f5f08" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39425 } }
I20260928 05:17:41.183120 5718 raft_consensus.cc:388] T 00000000000000000000000000000000 P fbdcf043ba074ba894d1d04b323f5f08 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:17:41.183233 5718 raft_consensus.cc:482] T 00000000000000000000000000000000 P fbdcf043ba074ba894d1d04b323f5f08 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:17:41.183375 5718 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fbdcf043ba074ba894d1d04b323f5f08 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:17:41.185391 5718 raft_consensus.cc:504] T 00000000000000000000000000000000 P fbdcf043ba074ba894d1d04b323f5f08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbdcf043ba074ba894d1d04b323f5f08" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39425 } }
I20260928 05:17:41.185824 5718 leader_election.cc:302] T 00000000000000000000000000000000 P fbdcf043ba074ba894d1d04b323f5f08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fbdcf043ba074ba894d1d04b323f5f08; no voters:
I20260928 05:17:41.186335 5718 leader_election.cc:288] T 00000000000000000000000000000000 P fbdcf043ba074ba894d1d04b323f5f08 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:17:41.186455 5723 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fbdcf043ba074ba894d1d04b323f5f08 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:17:41.187623 5723 raft_consensus.cc:686] T 00000000000000000000000000000000 P fbdcf043ba074ba894d1d04b323f5f08 [term 1 LEADER]: Becoming Leader. State: Replica: fbdcf043ba074ba894d1d04b323f5f08, State: Running, Role: LEADER
I20260928 05:17:41.188066 5723 consensus_queue.cc:237] T 00000000000000000000000000000000 P fbdcf043ba074ba894d1d04b323f5f08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbdcf043ba074ba894d1d04b323f5f08" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39425 } }
I20260928 05:17:41.188254 5718 sys_catalog.cc:564] T 00000000000000000000000000000000 P fbdcf043ba074ba894d1d04b323f5f08 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:17:41.190821 5724 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbdcf043ba074ba894d1d04b323f5f08 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fbdcf043ba074ba894d1d04b323f5f08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbdcf043ba074ba894d1d04b323f5f08" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39425 } } }
I20260928 05:17:41.191162 5724 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbdcf043ba074ba894d1d04b323f5f08 [sys.catalog]: This master's current role is: LEADER
I20260928 05:17:41.191588 5725 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbdcf043ba074ba894d1d04b323f5f08 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fbdcf043ba074ba894d1d04b323f5f08. Latest consensus state: current_term: 1 leader_uuid: "fbdcf043ba074ba894d1d04b323f5f08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbdcf043ba074ba894d1d04b323f5f08" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39425 } } }
I20260928 05:17:41.192912 5725 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbdcf043ba074ba894d1d04b323f5f08 [sys.catalog]: This master's current role is: LEADER
I20260928 05:17:41.195230 5730 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:17:41.201151 5730 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:17:41.210166 5730 catalog_manager.cc:1347] Generated new cluster ID: b05b3728f29547e3b146a83663783e68
I20260928 05:17:41.210273 5730 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:17:41.223671 5730 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:17:41.224893 5730 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:17:41.240036 5730 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fbdcf043ba074ba894d1d04b323f5f08: Generated new TSK 0
I20260928 05:17:41.240868 5730 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 05:17:41.277695 5748 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260928 05:17:41.277797 5748 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20260928 05:17:41.279019 5747 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:17:41.259666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39425 (local address 127.0.0.1:46054)
0928 05:17:41.260199 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:41.260214 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:41.260327 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:41.260736 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:41.260739 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:41.260747 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0928 05:17:41.261017 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:41.261026 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:41.262346 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:41.262350 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:41.263487 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:41.263498 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:41.264457 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:41.264496 (+ 39us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:41.264646 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:41.264679 (+ 33us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 05:17:41.278559 (+ 13880us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37179/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 05:17:41.278745 (+ 186us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:39425: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37179/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":295,"threads_started":1}
I20260928 05:17:41.279021 5721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:41.259821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46054 (local address 127.0.97.62:39425)
0928 05:17:41.260024 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:41.260030 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:41.260287 (+ 257us) server_negotiation.cc:407] Connection header received
0928 05:17:41.260491 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:41.260496 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:41.260585 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:41.260736 (+ 151us) server_negotiation.cc:226] Negotiated authn=JWT
0928 05:17:41.261146 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:41.262228 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:41.263613 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:41.264308 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:41.264440 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:41.264951 (+ 511us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 05:17:41.278342 (+ 13391us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37179/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 05:17:41.278705 (+ 363us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46054: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37179/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":70}
W20260928 05:17:41.279140 5721 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46054: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37179/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260928 05:17:41.280085 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 5652
I20260928 05:17:41.281299 5713 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:17:41.370518 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 5652
2026-09-28T05:17:41Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (675 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260928 05:17:41.422536 388 mini_oidc.cc:152] Starting JWKS server
I20260928 05:17:41.424237 388 webserver.cc:460] Webserver started at https://127.0.0.1:45739/ using document root <none> and password file <none>
I20260928 05:17:41.424432 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 05:17:41.425091 388 webserver.cc:460] Webserver started at http://127.0.0.1:39863/ using document root <none> and password file <none>
2026-09-28T05:17:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:17:41Z Disabled control of system clock
I20260928 05:17:41.447587 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790572622576160-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790572622576160-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790572622576160-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39009
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:40893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39009
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45739/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 05:17:41.746407 5755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:17:41.746672 5755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:17:41.746762 5755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:17:41.753539 5755 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:17:41.753629 5755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:17:41.753666 5755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:17:41.753719 5755 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:17:41.753760 5755 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 05:17:41.753816 5755 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 05:17:41.753854 5755 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45739/jwks.json
I20260928 05:17:41.762332 5755 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790572622576160-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790572622576160-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39009
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45739/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790572622576160-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:17:41.763445 5755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:17:41.765144 5755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:17:41.775478 5760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:41.776088 5763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:41.776115 5761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:17:41.776671 5755 server_base.cc:1034] running on GCE node
I20260928 05:17:41.777371 5755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:17:41.778666 5755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:17:41.779863 5755 hybrid_clock.cc:648] HybridClock initialized: now 1790572661779828 us; error 25 us; skew 500 ppm
I20260928 05:17:41.780172 5755 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 05:17:41.782227 5755 webserver.cc:460] Webserver started at http://127.0.97.62:38943/ using document root <none> and password file <none>
I20260928 05:17:41.782841 5755 fs_manager.cc:359] Metadata directory not provided
I20260928 05:17:41.782931 5755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:17:41.783227 5755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:17:41.785959 5755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790572622576160-388-0/minicluster-data/master-0/data/instance:
uuid: "caf53609393246eab2406132ae8e4da2"
format_stamp: "Formatted at 2026-09-28 05:17:41 on dist-test-slave-whsn"
server_key: "4acfa0289f89271d90e2bfbbf23f31d2"
server_key_iv: "8677b4beb86d6cf8c845814dd5bf2f40"
server_key_version: "encryptionkey@0"
I20260928 05:17:41.786690 5755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790572622576160-388-0/minicluster-data/master-0/wal/instance:
uuid: "caf53609393246eab2406132ae8e4da2"
format_stamp: "Formatted at 2026-09-28 05:17:41 on dist-test-slave-whsn"
server_key: "4acfa0289f89271d90e2bfbbf23f31d2"
server_key_iv: "8677b4beb86d6cf8c845814dd5bf2f40"
server_key_version: "encryptionkey@0"
I20260928 05:17:41.791412 5755 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260928 05:17:41.794696 5769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:41.795913 5755 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 05:17:41.796056 5755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790572622576160-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790572622576160-388-0/minicluster-data/master-0/wal
uuid: "caf53609393246eab2406132ae8e4da2"
format_stamp: "Formatted at 2026-09-28 05:17:41 on dist-test-slave-whsn"
server_key: "4acfa0289f89271d90e2bfbbf23f31d2"
server_key_iv: "8677b4beb86d6cf8c845814dd5bf2f40"
server_key_version: "encryptionkey@0"
I20260928 05:17:41.796238 5755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790572622576160-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790572622576160-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790572622576160-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:17:41.819976 5755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:17:41.820783 5755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:17:41.821058 5755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:17:41.837739 5755 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39009
I20260928 05:17:41.837749 5820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39009 every 8 connection(s)
I20260928 05:17:41.839010 5755 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:17:41.840675 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 5755
I20260928 05:17:41.840844 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790572622576160-388-0/minicluster-data/master-0/wal/instance
I20260928 05:17:41.841481 388 external_mini_cluster.cc:1428] Setting key 60e58a02b5a30d37bac89591d8151bf8
I20260928 05:17:41.842908 5821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:17:41.853751 5824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:41.843273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48440 (local address 127.0.97.62:39009)
0928 05:17:41.844191 (+ 918us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:41.844203 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:41.844241 (+ 38us) server_negotiation.cc:407] Connection header received
0928 05:17:41.845833 (+ 1592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:41.845855 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:41.846303 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:41.846667 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:41.847171 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:41.848462 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:41.849755 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:41.850378 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:41.850547 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:41.851621 (+ 1074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:41.851628 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:41.851643 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:17:41.852212 (+ 569us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:41.852281 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:17:41.852300 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:41.852397 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:41.852600 (+ 203us) server_negotiation.cc:299] Negotiation successful
0928 05:17:41.852891 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":326,"threads_started":1}
I20260928 05:17:41.854044 5821 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P caf53609393246eab2406132ae8e4da2: Bootstrap starting.
I20260928 05:17:41.857230 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 05:17:41.857406 5821 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P caf53609393246eab2406132ae8e4da2: Neither blocks nor log segments found. Creating new log.
I20260928 05:17:41.858294 5821 log.cc:824] T 00000000000000000000000000000000 P caf53609393246eab2406132ae8e4da2: Log is configured to *not* fsync() on all Append() calls
I20260928 05:17:41.860822 5821 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P caf53609393246eab2406132ae8e4da2: No bootstrap required, opened a new log
I20260928 05:17:41.867676 5821 raft_consensus.cc:348] T 00000000000000000000000000000000 P caf53609393246eab2406132ae8e4da2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caf53609393246eab2406132ae8e4da2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39009 } }
I20260928 05:17:41.868024 5821 raft_consensus.cc:374] T 00000000000000000000000000000000 P caf53609393246eab2406132ae8e4da2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:17:41.868127 5821 raft_consensus.cc:729] T 00000000000000000000000000000000 P caf53609393246eab2406132ae8e4da2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: caf53609393246eab2406132ae8e4da2, State: Initialized, Role: FOLLOWER
I20260928 05:17:41.868664 5821 consensus_queue.cc:260] T 00000000000000000000000000000000 P caf53609393246eab2406132ae8e4da2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caf53609393246eab2406132ae8e4da2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39009 } }
I20260928 05:17:41.868875 5821 raft_consensus.cc:388] T 00000000000000000000000000000000 P caf53609393246eab2406132ae8e4da2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:17:41.868999 5821 raft_consensus.cc:482] T 00000000000000000000000000000000 P caf53609393246eab2406132ae8e4da2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:17:41.869138 5821 raft_consensus.cc:3037] T 00000000000000000000000000000000 P caf53609393246eab2406132ae8e4da2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:17:41.871050 5821 raft_consensus.cc:504] T 00000000000000000000000000000000 P caf53609393246eab2406132ae8e4da2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caf53609393246eab2406132ae8e4da2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39009 } }
I20260928 05:17:41.871496 5821 leader_election.cc:302] T 00000000000000000000000000000000 P caf53609393246eab2406132ae8e4da2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: caf53609393246eab2406132ae8e4da2; no voters:
I20260928 05:17:41.871987 5821 leader_election.cc:288] T 00000000000000000000000000000000 P caf53609393246eab2406132ae8e4da2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:17:41.872120 5826 raft_consensus.cc:2781] T 00000000000000000000000000000000 P caf53609393246eab2406132ae8e4da2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:17:41.872745 5826 raft_consensus.cc:686] T 00000000000000000000000000000000 P caf53609393246eab2406132ae8e4da2 [term 1 LEADER]: Becoming Leader. State: Replica: caf53609393246eab2406132ae8e4da2, State: Running, Role: LEADER
I20260928 05:17:41.873275 5826 consensus_queue.cc:237] T 00000000000000000000000000000000 P caf53609393246eab2406132ae8e4da2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caf53609393246eab2406132ae8e4da2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39009 } }
I20260928 05:17:41.873850 5821 sys_catalog.cc:564] T 00000000000000000000000000000000 P caf53609393246eab2406132ae8e4da2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:17:41.875972 5827 sys_catalog.cc:455] T 00000000000000000000000000000000 P caf53609393246eab2406132ae8e4da2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "caf53609393246eab2406132ae8e4da2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caf53609393246eab2406132ae8e4da2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39009 } } }
I20260928 05:17:41.876232 5828 sys_catalog.cc:455] T 00000000000000000000000000000000 P caf53609393246eab2406132ae8e4da2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader caf53609393246eab2406132ae8e4da2. Latest consensus state: current_term: 1 leader_uuid: "caf53609393246eab2406132ae8e4da2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caf53609393246eab2406132ae8e4da2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39009 } } }
I20260928 05:17:41.876434 5827 sys_catalog.cc:458] T 00000000000000000000000000000000 P caf53609393246eab2406132ae8e4da2 [sys.catalog]: This master's current role is: LEADER
I20260928 05:17:41.876546 5828 sys_catalog.cc:458] T 00000000000000000000000000000000 P caf53609393246eab2406132ae8e4da2 [sys.catalog]: This master's current role is: LEADER
W20260928 05:17:41.883347 5842 catalog_manager.cc:1558] T 00000000000000000000000000000000 P caf53609393246eab2406132ae8e4da2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 05:17:41.883466 5842 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 05:17:41.883570 5835 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:17:41.884928 5835 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:17:41.891932 5835 catalog_manager.cc:1347] Generated new cluster ID: e39fb583870f4f6c8b004eae239bc0a9
I20260928 05:17:41.892035 5835 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:17:41.907914 5835 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:17:41.909806 5835 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:17:41.917065 5835 catalog_manager.cc:5868] T 00000000000000000000000000000000 P caf53609393246eab2406132ae8e4da2: Generated new TSK 0
I20260928 05:17:41.918064 5835 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 05:17:41.948302 5824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:41.941029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48454 (local address 127.0.97.62:39009)
0928 05:17:41.941220 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:41.941227 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:41.941591 (+ 364us) server_negotiation.cc:407] Connection header received
0928 05:17:41.941801 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:41.941805 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:41.941868 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:41.941984 (+ 116us) server_negotiation.cc:226] Negotiated authn=JWT
0928 05:17:41.942460 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:41.943142 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:41.945096 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:41.945525 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:41.945611 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:41.946151 (+ 540us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 05:17:41.947742 (+ 1591us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45739/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 05:17:41.948021 (+ 279us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48454: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45739/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":53}
W20260928 05:17:41.948513 5850 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:17:41.940914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39009 (local address 127.0.0.1:48454)
0928 05:17:41.941494 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:41.941513 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:41.941654 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:41.942009 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:41.942015 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:41.942028 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0928 05:17:41.942332 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:41.942343 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:41.943273 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:41.943280 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:41.944963 (+ 1683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:41.944978 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:41.945628 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:41.945695 (+ 67us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:41.945876 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:41.945918 (+ 42us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 05:17:41.947911 (+ 1993us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45739/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 05:17:41.948123 (+ 212us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:39009: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45739/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":297,"threads_started":1}
W20260928 05:17:41.948758 5824 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48454: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45739/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
I20260928 05:17:41.949241 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 5755
I20260928 05:17:41.950300 5816 generic_service.cc:196] Checking for leaks (request via RPC)
E20260928 05:17:41.965209 5851 webserver.cc:533] Webserver: sslize failed:
E20260928 05:17:41.965308 5851 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 05:17:42.041234 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 5755
2026-09-28T05:17:42Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (668 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-28T05:17:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:17:42Z Disabled control of system clock
I20260928 05:17:42.112532 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790572622576160-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790572622576160-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790572622576160-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40723
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:44745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40723
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790572622576160-388-0/testchainca.cert with env {}
W20260928 05:17:42.404419 5856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:17:42.404712 5856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:17:42.404819 5856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:17:42.411836 5856 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:17:42.411942 5856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:17:42.411993 5856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:17:42.412050 5856 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:17:42.412098 5856 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 05:17:42.412163 5856 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 05:17:42.412209 5856 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260928 05:17:42.421837 5856 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790572622576160-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790572622576160-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40723
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790572622576160-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790572622576160-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:17:42.423003 5856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:17:42.424237 5856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:17:42.435568 5861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:42.435653 5862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:42.435652 5864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:17:42.436364 5856 server_base.cc:1034] running on GCE node
I20260928 05:17:42.437150 5856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:17:42.438441 5856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:17:42.439639 5856 hybrid_clock.cc:648] HybridClock initialized: now 1790572662439591 us; error 36 us; skew 500 ppm
I20260928 05:17:42.439972 5856 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 05:17:42.442157 5856 webserver.cc:460] Webserver started at http://127.0.97.62:42285/ using document root <none> and password file <none>
I20260928 05:17:42.442756 5856 fs_manager.cc:359] Metadata directory not provided
I20260928 05:17:42.442854 5856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:17:42.443145 5856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:17:42.445792 5856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790572622576160-388-0/minicluster-data/master-0/data/instance:
uuid: "fa9045ac71914612a63b8b63587225b8"
format_stamp: "Formatted at 2026-09-28 05:17:42 on dist-test-slave-whsn"
server_key: "10020709704314eb5c6c62672076b77a"
server_key_iv: "1064b6855df355dbb77b02547a362841"
server_key_version: "encryptionkey@0"
I20260928 05:17:42.446506 5856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790572622576160-388-0/minicluster-data/master-0/wal/instance:
uuid: "fa9045ac71914612a63b8b63587225b8"
format_stamp: "Formatted at 2026-09-28 05:17:42 on dist-test-slave-whsn"
server_key: "10020709704314eb5c6c62672076b77a"
server_key_iv: "1064b6855df355dbb77b02547a362841"
server_key_version: "encryptionkey@0"
I20260928 05:17:42.450961 5856 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 05:17:42.454023 5870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:42.455224 5856 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 05:17:42.455366 5856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790572622576160-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790572622576160-388-0/minicluster-data/master-0/wal
uuid: "fa9045ac71914612a63b8b63587225b8"
format_stamp: "Formatted at 2026-09-28 05:17:42 on dist-test-slave-whsn"
server_key: "10020709704314eb5c6c62672076b77a"
server_key_iv: "1064b6855df355dbb77b02547a362841"
server_key_version: "encryptionkey@0"
I20260928 05:17:42.455540 5856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790572622576160-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790572622576160-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790572622576160-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:17:42.470299 5856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:17:42.471010 5856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:17:42.471240 5856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:17:42.486812 5856 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40723
I20260928 05:17:42.486832 5921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40723 every 8 connection(s)
I20260928 05:17:42.488013 5856 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:17:42.491497 5922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:17:42.496377 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 5856
I20260928 05:17:42.496603 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790572622576160-388-0/minicluster-data/master-0/wal/instance
I20260928 05:17:42.497439 388 external_mini_cluster.cc:1428] Setting key 3a282d235a693ec17646484d0a5c9d50
I20260928 05:17:42.505105 5922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa9045ac71914612a63b8b63587225b8: Bootstrap starting.
I20260928 05:17:42.508061 5922 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fa9045ac71914612a63b8b63587225b8: Neither blocks nor log segments found. Creating new log.
I20260928 05:17:42.509076 5922 log.cc:824] T 00000000000000000000000000000000 P fa9045ac71914612a63b8b63587225b8: Log is configured to *not* fsync() on all Append() calls
I20260928 05:17:42.511562 5922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa9045ac71914612a63b8b63587225b8: No bootstrap required, opened a new log
I20260928 05:17:42.518230 5922 raft_consensus.cc:348] T 00000000000000000000000000000000 P fa9045ac71914612a63b8b63587225b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa9045ac71914612a63b8b63587225b8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40723 } }
I20260928 05:17:42.518554 5922 raft_consensus.cc:374] T 00000000000000000000000000000000 P fa9045ac71914612a63b8b63587225b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:17:42.518642 5922 raft_consensus.cc:729] T 00000000000000000000000000000000 P fa9045ac71914612a63b8b63587225b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa9045ac71914612a63b8b63587225b8, State: Initialized, Role: FOLLOWER
I20260928 05:17:42.519109 5925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:42.499768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34730 (local address 127.0.97.62:40723)
0928 05:17:42.504553 (+ 4785us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:42.504569 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:42.504695 (+ 126us) server_negotiation.cc:407] Connection header received
0928 05:17:42.506650 (+ 1955us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:42.506672 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:42.507129 (+ 457us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:42.507488 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:42.508044 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:42.509456 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:42.510903 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:42.511588 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:42.511761 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:42.516800 (+ 5039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:42.516808 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:42.516822 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:17:42.517403 (+ 581us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:42.517482 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:17:42.517502 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:42.517607 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:42.517813 (+ 206us) server_negotiation.cc:299] Negotiation successful
0928 05:17:42.518099 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4227,"thread_start_us":3979,"threads_started":1}
I20260928 05:17:42.519168 5922 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa9045ac71914612a63b8b63587225b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa9045ac71914612a63b8b63587225b8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40723 } }
I20260928 05:17:42.519418 5922 raft_consensus.cc:388] T 00000000000000000000000000000000 P fa9045ac71914612a63b8b63587225b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:17:42.519549 5922 raft_consensus.cc:482] T 00000000000000000000000000000000 P fa9045ac71914612a63b8b63587225b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:17:42.519722 5922 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa9045ac71914612a63b8b63587225b8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:17:42.522497 5922 raft_consensus.cc:504] T 00000000000000000000000000000000 P fa9045ac71914612a63b8b63587225b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa9045ac71914612a63b8b63587225b8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40723 } }
I20260928 05:17:42.522876 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 05:17:42.523007 5922 leader_election.cc:302] T 00000000000000000000000000000000 P fa9045ac71914612a63b8b63587225b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fa9045ac71914612a63b8b63587225b8; no voters:
I20260928 05:17:42.523536 5922 leader_election.cc:288] T 00000000000000000000000000000000 P fa9045ac71914612a63b8b63587225b8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:17:42.523685 5927 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fa9045ac71914612a63b8b63587225b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:17:42.524612 5927 raft_consensus.cc:686] T 00000000000000000000000000000000 P fa9045ac71914612a63b8b63587225b8 [term 1 LEADER]: Becoming Leader. State: Replica: fa9045ac71914612a63b8b63587225b8, State: Running, Role: LEADER
I20260928 05:17:42.525254 5927 consensus_queue.cc:237] T 00000000000000000000000000000000 P fa9045ac71914612a63b8b63587225b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa9045ac71914612a63b8b63587225b8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40723 } }
I20260928 05:17:42.525491 5922 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa9045ac71914612a63b8b63587225b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:17:42.527668 5928 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa9045ac71914612a63b8b63587225b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fa9045ac71914612a63b8b63587225b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa9045ac71914612a63b8b63587225b8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40723 } } }
I20260928 05:17:42.527997 5928 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa9045ac71914612a63b8b63587225b8 [sys.catalog]: This master's current role is: LEADER
I20260928 05:17:42.529683 5932 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:17:42.531155 5929 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa9045ac71914612a63b8b63587225b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa9045ac71914612a63b8b63587225b8. Latest consensus state: current_term: 1 leader_uuid: "fa9045ac71914612a63b8b63587225b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa9045ac71914612a63b8b63587225b8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40723 } } }
I20260928 05:17:42.531421 5929 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa9045ac71914612a63b8b63587225b8 [sys.catalog]: This master's current role is: LEADER
I20260928 05:17:42.534183 5932 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:17:42.543355 5932 catalog_manager.cc:1347] Generated new cluster ID: a9e1d9a3b8bd49f898b82d23cf58fc50
I20260928 05:17:42.543442 5932 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:17:42.563002 5932 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:17:42.564172 5932 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:17:42.574548 5932 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fa9045ac71914612a63b8b63587225b8: Generated new TSK 0
I20260928 05:17:42.575306 5932 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 05:17:42.623005 5925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:42.587173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34742 (local address 127.0.97.62:40723)
0928 05:17:42.587338 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:42.587345 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:42.587815 (+ 470us) server_negotiation.cc:407] Connection header received
0928 05:17:42.588017 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:42.588021 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:42.588085 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:42.588198 (+ 113us) server_negotiation.cc:226] Negotiated authn=JWT
0928 05:17:42.588675 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:42.589433 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:42.591410 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:42.591816 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:42.591908 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:42.592394 (+ 486us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 05:17:42.622438 (+ 30044us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 05:17:42.622745 (+ 307us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34742: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":51}
W20260928 05:17:42.623142 5925 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34742: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260928 05:17:42.623720 5951 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:17:42.587158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40723 (local address 127.0.0.1:34742)
0928 05:17:42.587713 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:42.587733 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:42.587878 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:42.588224 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:42.588229 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:42.588241 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0928 05:17:42.588545 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:42.588556 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:42.589559 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:42.589566 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:42.591290 (+ 1724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:42.591304 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:42.591917 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:42.591965 (+ 48us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:42.592144 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:42.592185 (+ 41us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 05:17:42.622705 (+ 30520us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 05:17:42.623015 (+ 310us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:40723: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":284,"threads_started":1}
I20260928 05:17:42.625290 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 5856
I20260928 05:17:42.626550 5917 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:17:42.716665 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 5856
2026-09-28T05:17:42Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (655 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (373 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (378 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:17:43 dist-test-slave-whsn krb5kdc[5966](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:17:43 dist-test-slave-whsn krb5kdc[5966](info): set up 2 sockets
Sep 28 05:17:43 dist-test-slave-whsn krb5kdc[5966](info): commencing operation
krb5kdc: starting...
W20260928 05:17:44.457135 388 mini_kdc.cc:121] Time spent starting KDC: real 0.957s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/krb5kdc/test-user.keytab.
Sep 28 05:17:44 dist-test-slave-whsn krb5kdc[5966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572664, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:17:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:17:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 05:17:44.625133 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41219
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41219
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:17:44.915210 5979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:17:44.915467 5979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:17:44.915588 5979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:17:44.922348 5979 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:17:44.922443 5979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:17:44.922513 5979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:17:44.922580 5979 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:17:44.922639 5979 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:17:44.931773 5979 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41219
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:17:44.932946 5979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:17:44.934150 5979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:17:44.944273 5985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:44.944367 5987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:44.944245 5984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:17:44.945360 5979 server_base.cc:1034] running on GCE node
I20260928 05:17:44.945994 5979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:17:44.947175 5979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:17:44.948355 5979 hybrid_clock.cc:648] HybridClock initialized: now 1790572664948314 us; error 41 us; skew 500 ppm
I20260928 05:17:44.948625 5979 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:17:44 dist-test-slave-whsn krb5kdc[5966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572664, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:17:44.951380 5979 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 05:17:44.952597 5979 webserver.cc:460] Webserver started at http://127.0.97.62:39775/ using document root <none> and password file <none>
I20260928 05:17:44.953223 5979 fs_manager.cc:359] Metadata directory not provided
I20260928 05:17:44.953326 5979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:17:44.953609 5979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:17:44.956228 5979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/minicluster-data/master-0/data/instance:
uuid: "614ffa52a9fe443fa982adcef6d28b99"
format_stamp: "Formatted at 2026-09-28 05:17:44 on dist-test-slave-whsn"
server_key: "dfcfeac59970f206d01c2126e96877ab"
server_key_iv: "75da37cf0e032f68f05e182e15f48dbe"
server_key_version: "encryptionkey@0"
I20260928 05:17:44.957042 5979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/minicluster-data/master-0/wal/instance:
uuid: "614ffa52a9fe443fa982adcef6d28b99"
format_stamp: "Formatted at 2026-09-28 05:17:44 on dist-test-slave-whsn"
server_key: "dfcfeac59970f206d01c2126e96877ab"
server_key_iv: "75da37cf0e032f68f05e182e15f48dbe"
server_key_version: "encryptionkey@0"
I20260928 05:17:44.961599 5979 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 05:17:44.964519 5994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:44.965694 5979 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 05:17:44.965835 5979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/minicluster-data/master-0/wal
uuid: "614ffa52a9fe443fa982adcef6d28b99"
format_stamp: "Formatted at 2026-09-28 05:17:44 on dist-test-slave-whsn"
server_key: "dfcfeac59970f206d01c2126e96877ab"
server_key_iv: "75da37cf0e032f68f05e182e15f48dbe"
server_key_version: "encryptionkey@0"
I20260928 05:17:44.965991 5979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:17:44.994621 5979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:17:44.998492 5979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:17:44.998731 5979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:17:45.014587 5979 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41219
I20260928 05:17:45.014624 6045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41219 every 8 connection(s)
I20260928 05:17:45.015827 5979 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:17:45.018910 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 5979
I20260928 05:17:45.019143 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/minicluster-data/master-0/wal/instance
I20260928 05:17:45.019534 6046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:17:45.019874 388 external_mini_cluster.cc:1428] Setting key f5e5c0efb35ad82cfa360b0cc3425d81
I20260928 05:17:45.032891 6046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 614ffa52a9fe443fa982adcef6d28b99: Bootstrap starting.
Sep 28 05:17:45 dist-test-slave-whsn krb5kdc[5966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572664, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:17:45.036072 6046 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 614ffa52a9fe443fa982adcef6d28b99: Neither blocks nor log segments found. Creating new log.
I20260928 05:17:45.037276 6046 log.cc:824] T 00000000000000000000000000000000 P 614ffa52a9fe443fa982adcef6d28b99: Log is configured to *not* fsync() on all Append() calls
I20260928 05:17:45.040256 6046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 614ffa52a9fe443fa982adcef6d28b99: No bootstrap required, opened a new log
I20260928 05:17:45.042243 6049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:45.021743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42758 (local address 127.0.97.62:41219)
0928 05:17:45.022477 (+ 734us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:45.022488 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:45.022525 (+ 37us) server_negotiation.cc:407] Connection header received
0928 05:17:45.024047 (+ 1522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:45.024068 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:45.024439 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:45.024768 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:45.026381 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:45.027629 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:45.029239 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:45.029903 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:45.030030 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:45.034517 (+ 4487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:45.034525 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:45.034539 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:45.034582 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:45.037988 (+ 3406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:45.038725 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:45.038733 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:45.038740 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:45.038920 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:45.039308 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:45.039313 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:45.039316 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:45.039929 (+ 613us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:45.040226 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:45.040549 (+ 323us) server_negotiation.cc:299] Negotiation successful
0928 05:17:45.041000 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":345,"threads_started":1}
I20260928 05:17:45.048426 6046 raft_consensus.cc:348] T 00000000000000000000000000000000 P 614ffa52a9fe443fa982adcef6d28b99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "614ffa52a9fe443fa982adcef6d28b99" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41219 } }
I20260928 05:17:45.048727 6046 raft_consensus.cc:374] T 00000000000000000000000000000000 P 614ffa52a9fe443fa982adcef6d28b99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:17:45.048794 6046 raft_consensus.cc:729] T 00000000000000000000000000000000 P 614ffa52a9fe443fa982adcef6d28b99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 614ffa52a9fe443fa982adcef6d28b99, State: Initialized, Role: FOLLOWER
I20260928 05:17:45.049310 6046 consensus_queue.cc:260] T 00000000000000000000000000000000 P 614ffa52a9fe443fa982adcef6d28b99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "614ffa52a9fe443fa982adcef6d28b99" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41219 } }
I20260928 05:17:45.049504 6046 raft_consensus.cc:388] T 00000000000000000000000000000000 P 614ffa52a9fe443fa982adcef6d28b99 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:17:45.049592 6046 raft_consensus.cc:482] T 00000000000000000000000000000000 P 614ffa52a9fe443fa982adcef6d28b99 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:17:45.049707 6046 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 614ffa52a9fe443fa982adcef6d28b99 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:17:45.051578 6046 raft_consensus.cc:504] T 00000000000000000000000000000000 P 614ffa52a9fe443fa982adcef6d28b99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "614ffa52a9fe443fa982adcef6d28b99" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41219 } }
I20260928 05:17:45.051960 6046 leader_election.cc:302] T 00000000000000000000000000000000 P 614ffa52a9fe443fa982adcef6d28b99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 614ffa52a9fe443fa982adcef6d28b99; no voters:
I20260928 05:17:45.052402 6046 leader_election.cc:288] T 00000000000000000000000000000000 P 614ffa52a9fe443fa982adcef6d28b99 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:17:45.052520 6051 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 614ffa52a9fe443fa982adcef6d28b99 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:17:45.053208 6051 raft_consensus.cc:686] T 00000000000000000000000000000000 P 614ffa52a9fe443fa982adcef6d28b99 [term 1 LEADER]: Becoming Leader. State: Replica: 614ffa52a9fe443fa982adcef6d28b99, State: Running, Role: LEADER
I20260928 05:17:45.053723 6051 consensus_queue.cc:237] T 00000000000000000000000000000000 P 614ffa52a9fe443fa982adcef6d28b99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "614ffa52a9fe443fa982adcef6d28b99" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41219 } }
I20260928 05:17:45.054379 6046 sys_catalog.cc:564] T 00000000000000000000000000000000 P 614ffa52a9fe443fa982adcef6d28b99 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:17:45.056149 6052 sys_catalog.cc:455] T 00000000000000000000000000000000 P 614ffa52a9fe443fa982adcef6d28b99 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "614ffa52a9fe443fa982adcef6d28b99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "614ffa52a9fe443fa982adcef6d28b99" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41219 } } }
I20260928 05:17:45.056537 6052 sys_catalog.cc:458] T 00000000000000000000000000000000 P 614ffa52a9fe443fa982adcef6d28b99 [sys.catalog]: This master's current role is: LEADER
I20260928 05:17:45.056212 6053 sys_catalog.cc:455] T 00000000000000000000000000000000 P 614ffa52a9fe443fa982adcef6d28b99 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 614ffa52a9fe443fa982adcef6d28b99. Latest consensus state: current_term: 1 leader_uuid: "614ffa52a9fe443fa982adcef6d28b99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "614ffa52a9fe443fa982adcef6d28b99" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41219 } } }
I20260928 05:17:45.056807 6053 sys_catalog.cc:458] T 00000000000000000000000000000000 P 614ffa52a9fe443fa982adcef6d28b99 [sys.catalog]: This master's current role is: LEADER
I20260928 05:17:45.058527 6059 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:17:45.063596 6059 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:17:45.071123 6059 catalog_manager.cc:1347] Generated new cluster ID: 0c3e1065b2464ed5b51f07083fc3fd53
I20260928 05:17:45.071223 6059 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:17:45.088220 6059 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:17:45.089489 6059 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:17:45.106236 6059 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 614ffa52a9fe443fa982adcef6d28b99: Generated new TSK 0
I20260928 05:17:45.107000 6059 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260928 05:17:45.152215 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41219
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:17:45.455495 6072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:17:45.455775 6072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:17:45.455875 6072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:17:45.462905 6072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:17:45.463011 6072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:17:45.463127 6072 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 05:17:45.472464 6072 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41219
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:17:45.473667 6072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:17:45.474865 6072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:17:45.486302 6077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:45.486531 6080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:45.486562 6078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:17:45.487408 6072 server_base.cc:1034] running on GCE node
I20260928 05:17:45.487915 6072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:17:45.488730 6072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:17:45.489915 6072 hybrid_clock.cc:648] HybridClock initialized: now 1790572665489858 us; error 54 us; skew 500 ppm
I20260928 05:17:45.490200 6072 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:17:45 dist-test-slave-whsn krb5kdc[5966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572665, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:17:45.492949 6072 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260928 05:17:45.494292 6072 webserver.cc:460] Webserver started at http://127.0.97.1:38779/ using document root <none> and password file <none>
I20260928 05:17:45.494882 6072 fs_manager.cc:359] Metadata directory not provided
I20260928 05:17:45.494989 6072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:17:45.495278 6072 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:17:45.497977 6072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/minicluster-data/ts-0/data/instance:
uuid: "7fdb65f3e08947c5bce9a26b914e6b3e"
format_stamp: "Formatted at 2026-09-28 05:17:45 on dist-test-slave-whsn"
server_key: "90e25d45d50aa63bab73b24159d280dd"
server_key_iv: "6ea926c408129d32300f3c1faba4b1fe"
server_key_version: "encryptionkey@0"
I20260928 05:17:45.498692 6072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/minicluster-data/ts-0/wal/instance:
uuid: "7fdb65f3e08947c5bce9a26b914e6b3e"
format_stamp: "Formatted at 2026-09-28 05:17:45 on dist-test-slave-whsn"
server_key: "90e25d45d50aa63bab73b24159d280dd"
server_key_iv: "6ea926c408129d32300f3c1faba4b1fe"
server_key_version: "encryptionkey@0"
I20260928 05:17:45.503434 6072 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 05:17:45.506564 6087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:45.507714 6072 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 05:17:45.507844 6072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/minicluster-data/ts-0/wal
uuid: "7fdb65f3e08947c5bce9a26b914e6b3e"
format_stamp: "Formatted at 2026-09-28 05:17:45 on dist-test-slave-whsn"
server_key: "90e25d45d50aa63bab73b24159d280dd"
server_key_iv: "6ea926c408129d32300f3c1faba4b1fe"
server_key_version: "encryptionkey@0"
I20260928 05:17:45.508019 6072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:17:45.534791 6072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:17:45.538652 6072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:17:45.538903 6072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:17:45.539779 6072 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:17:45.541132 6072 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:17:45.541206 6072 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:45.541314 6072 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:17:45.541364 6072 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:45.569655 6072 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:36783
I20260928 05:17:45.569700 6199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:36783 every 8 connection(s)
I20260928 05:17:45.570832 6072 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/minicluster-data/ts-0/data/info.pb
I20260928 05:17:45.577571 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 6072
I20260928 05:17:45.577773 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/minicluster-data/ts-0/wal/instance
I20260928 05:17:45.578421 388 external_mini_cluster.cc:1428] Setting key bac8776fff208c118159986b73f8aaf7
Sep 28 05:17:45 dist-test-slave-whsn krb5kdc[5966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572665, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:17:45.590423 6202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:45.573159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52485 (local address 127.0.97.62:41219)
0928 05:17:45.573539 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:45.573543 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:45.574636 (+ 1093us) server_negotiation.cc:407] Connection header received
0928 05:17:45.576430 (+ 1794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:45.576435 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:45.576497 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:45.576599 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:45.578496 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:45.579449 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:45.581094 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:45.581478 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:45.581583 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:45.585074 (+ 3491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:45.585079 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:45.585082 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:45.585113 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:45.587737 (+ 2624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:45.588364 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:45.588369 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:45.588372 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:45.588505 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:45.588842 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:45.588846 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:45.588871 (+ 25us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:45.589192 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:45.589322 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:45.589895 (+ 573us) server_negotiation.cc:299] Negotiation successful
0928 05:17:45.590093 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":197,"threads_started":1}
I20260928 05:17:45.591136 6203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:45.573755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41219 (local address 127.0.97.1:52485)
0928 05:17:45.574479 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:45.574523 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:45.576185 (+ 1662us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:45.576942 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:45.576950 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:45.577593 (+ 643us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:17:45.578333 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:45.578351 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:45.579590 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:45.579596 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:45.580962 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:45.580973 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:45.581678 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:45.581686 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:45.581906 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:45.582492 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:17:45.582523 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:17:45.584844 (+ 2321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:17:45.587856 (+ 3012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:45.587862 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:45.587873 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:45.588247 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:45.588591 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:45.588594 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:45.588597 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:45.588751 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:45.589294 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:17:45.589299 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:17:45.589581 (+ 282us) client_negotiation.cc:769] Sending connection context
0928 05:17:45.589781 (+ 200us) client_negotiation.cc:240] Negotiation successful
0928 05:17:45.590114 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":294,"threads_started":1}
I20260928 05:17:45.592506 6200 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41219
I20260928 05:17:45.592780 6200 heartbeater.cc:464] Registering TS with master...
I20260928 05:17:45.593472 6200 heartbeater.cc:511] Master 127.0.97.62:41219 requested a full tablet report, sending...
I20260928 05:17:45.595376 6011 ts_manager.cc:194] Registered new tserver with Master: 7fdb65f3e08947c5bce9a26b914e6b3e (127.0.97.1:36783)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260928 05:17:45.596791 6011 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:52485
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260928 05:17:45.618038 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41219
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:17:45.916309 6206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:17:45.916561 6206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:17:45.916699 6206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:17:45.923585 6206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:17:45.923689 6206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:17:45.923839 6206 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 05:17:45.932888 6206 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41219
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:17:45.934010 6206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:17:45.935186 6206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:17:45.946900 6211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:45.946866 6212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:45.946928 6214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:17:45.947770 6206 server_base.cc:1034] running on GCE node
I20260928 05:17:45.948252 6206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:17:45.949071 6206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:17:45.950249 6206 hybrid_clock.cc:648] HybridClock initialized: now 1790572665950213 us; error 31 us; skew 500 ppm
I20260928 05:17:45.950522 6206 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:17:45 dist-test-slave-whsn krb5kdc[5966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572665, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:17:45.953217 6206 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260928 05:17:45.954483 6206 webserver.cc:460] Webserver started at http://127.0.97.2:38909/ using document root <none> and password file <none>
I20260928 05:17:45.955113 6206 fs_manager.cc:359] Metadata directory not provided
I20260928 05:17:45.955222 6206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:17:45.955472 6206 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:17:45.958122 6206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/minicluster-data/ts-1/data/instance:
uuid: "493a3bf03c604c29b3b81debbf8ba73b"
format_stamp: "Formatted at 2026-09-28 05:17:45 on dist-test-slave-whsn"
server_key: "1bc6eab846b603772c53a5072658a48f"
server_key_iv: "cf6cc07047a98905ccfa438d7d50983b"
server_key_version: "encryptionkey@0"
I20260928 05:17:45.958815 6206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/minicluster-data/ts-1/wal/instance:
uuid: "493a3bf03c604c29b3b81debbf8ba73b"
format_stamp: "Formatted at 2026-09-28 05:17:45 on dist-test-slave-whsn"
server_key: "1bc6eab846b603772c53a5072658a48f"
server_key_iv: "cf6cc07047a98905ccfa438d7d50983b"
server_key_version: "encryptionkey@0"
I20260928 05:17:45.963326 6206 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 05:17:45.966302 6221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:45.967474 6206 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 05:17:45.967624 6206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/minicluster-data/ts-1/wal
uuid: "493a3bf03c604c29b3b81debbf8ba73b"
format_stamp: "Formatted at 2026-09-28 05:17:45 on dist-test-slave-whsn"
server_key: "1bc6eab846b603772c53a5072658a48f"
server_key_iv: "cf6cc07047a98905ccfa438d7d50983b"
server_key_version: "encryptionkey@0"
I20260928 05:17:45.967785 6206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:17:46.006907 6206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:17:46.010532 6206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:17:46.010785 6206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:17:46.011685 6206 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:17:46.012921 6206 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:17:46.012995 6206 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:46.013108 6206 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:17:46.013178 6206 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:46.041852 6206 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43123
I20260928 05:17:46.041915 6333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43123 every 8 connection(s)
I20260928 05:17:46.042975 6206 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/minicluster-data/ts-1/data/info.pb
I20260928 05:17:46.043547 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 6206
I20260928 05:17:46.043725 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/minicluster-data/ts-1/wal/instance
I20260928 05:17:46.044242 388 external_mini_cluster.cc:1428] Setting key 31ecc0926c9c295d06798f2d0c728ea5
Sep 28 05:17:46 dist-test-slave-whsn krb5kdc[5966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572665, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260928 05:17:46.062650 6202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:46.045432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37531 (local address 127.0.97.62:41219)
0928 05:17:46.045585 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:46.045590 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:46.047041 (+ 1451us) server_negotiation.cc:407] Connection header received
0928 05:17:46.048730 (+ 1689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:46.048735 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:46.048810 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:46.048928 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:46.050279 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:46.051235 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:46.053354 (+ 2119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:46.053739 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:46.053825 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:46.057451 (+ 3626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:46.057457 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:46.057461 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:46.057511 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:46.059965 (+ 2454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:46.060593 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:46.060598 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:46.060600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:46.060714 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:46.061127 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:46.061132 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:46.061135 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:46.061371 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:46.061500 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:46.062116 (+ 616us) server_negotiation.cc:299] Negotiation successful
0928 05:17:46.062323 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 05:17:46.063562 6336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:46.046087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41219 (local address 127.0.97.2:37531)
0928 05:17:46.046869 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:46.046920 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:46.048499 (+ 1579us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:46.049106 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:46.049114 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:46.049599 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:17:46.050131 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:46.050148 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:46.051356 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:46.051362 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:46.053231 (+ 1869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:46.053246 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:46.053937 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:46.053945 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:46.054249 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:46.054888 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:17:46.054913 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:17:46.057255 (+ 2342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:17:46.060088 (+ 2833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:46.060097 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:46.060112 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:46.060487 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:46.060812 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:46.060817 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:46.060821 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:46.061031 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:46.061492 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:17:46.061500 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:17:46.061790 (+ 290us) client_negotiation.cc:769] Sending connection context
0928 05:17:46.062011 (+ 221us) client_negotiation.cc:240] Negotiation successful
0928 05:17:46.062476 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":324,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260928 05:17:46.064963 6334 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41219
I20260928 05:17:46.065238 6334 heartbeater.cc:464] Registering TS with master...
I20260928 05:17:46.065905 6334 heartbeater.cc:511] Master 127.0.97.62:41219 requested a full tablet report, sending...
I20260928 05:17:46.067273 6011 ts_manager.cc:194] Registered new tserver with Master: 493a3bf03c604c29b3b81debbf8ba73b (127.0.97.2:43123)
I20260928 05:17:46.068212 6011 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:37531
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260928 05:17:46.085516 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41219
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:17:46.381774 6339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:17:46.382020 6339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:17:46.382113 6339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:17:46.388916 6339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:17:46.389014 6339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:17:46.389123 6339 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 05:17:46.398131 6339 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41219
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:17:46.399257 6339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:17:46.400449 6339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:17:46.411990 6345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:46.412060 6344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:46.412034 6347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:17:46.413062 6339 server_base.cc:1034] running on GCE node
I20260928 05:17:46.413565 6339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:17:46.414403 6339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:17:46.415591 6339 hybrid_clock.cc:648] HybridClock initialized: now 1790572666415541 us; error 49 us; skew 500 ppm
I20260928 05:17:46.415855 6339 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:17:46 dist-test-slave-whsn krb5kdc[5966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572666, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:17:46.418848 6339 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260928 05:17:46.420146 6339 webserver.cc:460] Webserver started at http://127.0.97.3:44649/ using document root <none> and password file <none>
I20260928 05:17:46.420739 6339 fs_manager.cc:359] Metadata directory not provided
I20260928 05:17:46.420840 6339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:17:46.421144 6339 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:17:46.423857 6339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/minicluster-data/ts-2/data/instance:
uuid: "6a413f1e3285488c9ad746aa41157937"
format_stamp: "Formatted at 2026-09-28 05:17:46 on dist-test-slave-whsn"
server_key: "80249131067519af12928fd7996491e0"
server_key_iv: "3a3e30d41c015595fc4296c98302f152"
server_key_version: "encryptionkey@0"
I20260928 05:17:46.424583 6339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/minicluster-data/ts-2/wal/instance:
uuid: "6a413f1e3285488c9ad746aa41157937"
format_stamp: "Formatted at 2026-09-28 05:17:46 on dist-test-slave-whsn"
server_key: "80249131067519af12928fd7996491e0"
server_key_iv: "3a3e30d41c015595fc4296c98302f152"
server_key_version: "encryptionkey@0"
I20260928 05:17:46.429451 6339 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260928 05:17:46.432701 6354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:46.434024 6339 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:17:46.434173 6339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/minicluster-data/ts-2/wal
uuid: "6a413f1e3285488c9ad746aa41157937"
format_stamp: "Formatted at 2026-09-28 05:17:46 on dist-test-slave-whsn"
server_key: "80249131067519af12928fd7996491e0"
server_key_iv: "3a3e30d41c015595fc4296c98302f152"
server_key_version: "encryptionkey@0"
I20260928 05:17:46.434350 6339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:17:46.459235 6339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:17:46.463195 6339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:17:46.463474 6339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:17:46.464406 6339 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:17:46.465767 6339 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:17:46.465849 6339 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:46.465979 6339 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:17:46.466037 6339 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:46.498598 6339 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36531
I20260928 05:17:46.498663 6466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36531 every 8 connection(s)
I20260928 05:17:46.500170 6339 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/minicluster-data/ts-2/data/info.pb
I20260928 05:17:46.510425 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 6339
I20260928 05:17:46.510617 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/minicluster-data/ts-2/wal/instance
I20260928 05:17:46.511083 388 external_mini_cluster.cc:1428] Setting key aa0ebb1b2c5f338538b8a5fdb34ebbca
Sep 28 05:17:46 dist-test-slave-whsn krb5kdc[5966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572666, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:17:46.518395 6202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:46.502920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42289 (local address 127.0.97.62:41219)
0928 05:17:46.503119 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:46.503126 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:46.504263 (+ 1137us) server_negotiation.cc:407] Connection header received
0928 05:17:46.505962 (+ 1699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:46.505967 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:46.506034 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:46.506121 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:46.507430 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:46.508180 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:46.509621 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:46.510029 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:46.510147 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:46.513637 (+ 3490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:46.513643 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:46.513647 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:46.513685 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:46.515961 (+ 2276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:46.516541 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:46.516544 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:46.516546 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:46.516647 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:46.516987 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:46.516998 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:46.517000 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:46.517312 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:46.517426 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:46.518035 (+ 609us) server_negotiation.cc:299] Negotiation successful
0928 05:17:46.518176 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 05:17:46.519217 6469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:46.503414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41219 (local address 127.0.97.3:42289)
0928 05:17:46.504115 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:46.504155 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:46.505715 (+ 1560us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:46.506297 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:46.506304 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:46.506739 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:17:46.507272 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:46.507288 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:46.508305 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:46.508309 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:46.509508 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:46.509519 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:46.510251 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:46.510259 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:46.510545 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:46.511206 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:17:46.511232 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:17:46.513434 (+ 2202us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:17:46.516073 (+ 2639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:46.516079 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:46.516088 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:46.516455 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:46.516716 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:46.516718 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:46.516721 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:46.516898 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:46.517428 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:17:46.517434 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:17:46.517736 (+ 302us) client_negotiation.cc:769] Sending connection context
0928 05:17:46.517940 (+ 204us) client_negotiation.cc:240] Negotiation successful
0928 05:17:46.518252 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":285,"threads_started":1}
I20260928 05:17:46.520329 6467 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41219
I20260928 05:17:46.520555 6467 heartbeater.cc:464] Registering TS with master...
I20260928 05:17:46.521193 6467 heartbeater.cc:511] Master 127.0.97.62:41219 requested a full tablet report, sending...
I20260928 05:17:46.522326 6011 ts_manager.cc:194] Registered new tserver with Master: 6a413f1e3285488c9ad746aa41157937 (127.0.97.3:36531)
I20260928 05:17:46.523041 6011 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:42289
I20260928 05:17:46.525619 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/krb5kdc/test-admin.keytab.
Sep 28 05:17:46 dist-test-slave-whsn krb5kdc[5966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:17:46.549196 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260928 05:17:46.553365 6474 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:17:46.551390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36222 (local address 127.0.97.1:36783)
0928 05:17:46.551966 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:46.551973 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:46.551998 (+ 25us) server_negotiation.cc:407] Connection header received
0928 05:17:46.552053 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:46.552059 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:46.552259 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:46.552383 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:46.553082 (+ 699us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36222: BlockingRecv error: recv got EOF from 127.0.0.1:36222 (error 108)
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":278,"threads_started":1}
W20260928 05:17:46.553638 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/krb5kdc/krb5cc' of type FILE
Sep 28 05:17:46 dist-test-slave-whsn krb5kdc[5966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:17:46.555243 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 05:17:46 dist-test-slave-whsn krb5kdc[5966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:17:46.556489 388 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 05:17:46 dist-test-slave-whsn krb5kdc[5966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260928 05:17:46.569895 6474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:46.558006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36228 (local address 127.0.97.1:36783)
0928 05:17:46.558163 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:46.558168 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:46.558335 (+ 167us) server_negotiation.cc:407] Connection header received
0928 05:17:46.558485 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:46.558489 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:46.558546 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:46.558636 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:46.559382 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:46.560315 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:46.561488 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:46.561995 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:46.562085 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:46.565188 (+ 3103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:46.565198 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:46.565210 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:46.565239 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:46.567528 (+ 2289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:46.568058 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:46.568064 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:46.568066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:46.568169 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:46.568398 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:46.568400 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:46.568402 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:46.568730 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:46.569045 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:46.569335 (+ 290us) server_negotiation.cc:299] Negotiation successful
0928 05:17:46.569604 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260928 05:17:46.575304 6474 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:17:46.573873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36232 (local address 127.0.97.1:36783)
0928 05:17:46.574024 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:46.574030 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:46.574143 (+ 113us) server_negotiation.cc:407] Connection header received
0928 05:17:46.574274 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:46.574277 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:46.574339 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:46.574408 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:46.575120 (+ 712us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36232: BlockingRecv error: recv got EOF from 127.0.0.1:36232 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260928 05:17:46.575479 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/krb5kdc/krb5cc' of type FILE
Sep 28 05:17:46 dist-test-slave-whsn krb5kdc[5966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:17:46.576746 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 05:17:46 dist-test-slave-whsn krb5kdc[5966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:17:46.577857 388 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 05:17:46 dist-test-slave-whsn krb5kdc[5966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260928 05:17:46.590162 6474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:46.578872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36234 (local address 127.0.97.1:36783)
0928 05:17:46.579012 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:46.579017 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:46.579303 (+ 286us) server_negotiation.cc:407] Connection header received
0928 05:17:46.579429 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:46.579433 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:46.579478 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:46.579549 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:46.580336 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:46.581054 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:46.582190 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:46.582592 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:46.582700 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:46.585838 (+ 3138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:46.585842 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:46.585845 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:46.585873 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:46.588051 (+ 2178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:46.588562 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:46.588566 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:46.588568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:46.588667 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:46.588992 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:46.588995 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:46.588997 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:46.589243 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:46.589474 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:46.589728 (+ 254us) server_negotiation.cc:299] Negotiation successful
0928 05:17:46.589931 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260928 05:17:46.593698 6474 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:17:46.592292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36250 (local address 127.0.97.1:36783)
0928 05:17:46.592420 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:46.592424 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:46.592701 (+ 277us) server_negotiation.cc:407] Connection header received
0928 05:17:46.592804 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:46.592807 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:46.592899 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:46.592998 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:46.593503 (+ 505us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36250: BlockingRecv error: recv got EOF from 127.0.0.1:36250 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260928 05:17:46.593897 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572622576160-388-0/krb5kdc/krb5cc' of type FILE
Sep 28 05:17:46 dist-test-slave-whsn krb5kdc[5966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:17:46.595135 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 05:17:46 dist-test-slave-whsn krb5kdc[5966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:17:46.596244 388 init.cc:295] Successfully reacquired a new kerberos TGT
I20260928 05:17:46.599545 6200 heartbeater.cc:503] Master 127.0.97.62:41219 was elected leader, sending a full tablet report...
Sep 28 05:17:46 dist-test-slave-whsn krb5kdc[5966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260928 05:17:46.608557 6474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:46.597335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36262 (local address 127.0.97.1:36783)
0928 05:17:46.597502 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:46.597507 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:46.597713 (+ 206us) server_negotiation.cc:407] Connection header received
0928 05:17:46.597836 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:46.597841 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:46.597915 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:46.597990 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:46.598889 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:46.599561 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:46.600712 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:46.601162 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:46.601263 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:46.604211 (+ 2948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:46.604216 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:46.604219 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:46.604248 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:46.606499 (+ 2251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:46.607149 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:46.607153 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:46.607155 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:46.607299 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:46.607580 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:46.607583 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:46.607585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:46.607815 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:46.608039 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:46.608170 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 05:17:46.608312 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 05:17:46.609828 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 6072
I20260928 05:17:46.620177 6474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:46.610466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36272 (local address 127.0.97.1:36783)
0928 05:17:46.610674 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:46.610708 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:46.610948 (+ 240us) server_negotiation.cc:407] Connection header received
0928 05:17:46.611127 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:46.611132 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:46.611197 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:46.611285 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:46.612059 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:46.612706 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:46.613810 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:46.614265 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:46.614363 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:46.615982 (+ 1619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:46.615986 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:46.615989 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:46.616014 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:46.618116 (+ 2102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:46.618642 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:46.618646 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:46.618648 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:46.618761 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:46.619067 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:46.619071 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:46.619074 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:46.619282 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:46.619499 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:46.619758 (+ 259us) server_negotiation.cc:299] Negotiation successful
0928 05:17:46.619933 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 05:17:46.620679 6195 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:17:46.699244 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 6072
I20260928 05:17:46.712189 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 6206
Sep 28 05:17:46 dist-test-slave-whsn krb5kdc[5966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260928 05:17:46.725279 6487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:46.712962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55278 (local address 127.0.97.2:43123)
0928 05:17:46.713484 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:46.713502 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:46.713527 (+ 25us) server_negotiation.cc:407] Connection header received
0928 05:17:46.713584 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:46.713591 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:46.713803 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:46.713927 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:46.714764 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:46.715740 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:46.716917 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:46.717462 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:46.717570 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:46.720762 (+ 3192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:46.720772 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:46.720786 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:46.720823 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:46.723166 (+ 2343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:46.723689 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:46.723695 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:46.723697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:46.723797 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:46.724053 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:46.724055 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:46.724057 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:46.724336 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:46.724553 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:46.724756 (+ 203us) server_negotiation.cc:299] Negotiation successful
0928 05:17:46.724988 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":267,"threads_started":1}
I20260928 05:17:46.726469 6329 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:17:46.804996 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 6206
I20260928 05:17:46.816836 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 6339
Sep 28 05:17:46 dist-test-slave-whsn krb5kdc[5966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260928 05:17:46.831032 6489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:46.817827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49178 (local address 127.0.97.3:36531)
0928 05:17:46.818380 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:46.818386 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:46.818416 (+ 30us) server_negotiation.cc:407] Connection header received
0928 05:17:46.818473 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:46.818479 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:46.818682 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:46.818817 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:46.819822 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:46.820794 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:46.821974 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:46.822495 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:46.822602 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:46.826144 (+ 3542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:46.826154 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:46.826167 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:46.826195 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:46.828532 (+ 2337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:46.829234 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:46.829241 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:46.829243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:46.829349 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:46.829678 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:46.829683 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:46.829686 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:46.829995 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:46.830267 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:46.830524 (+ 257us) server_negotiation.cc:299] Negotiation successful
0928 05:17:46.830749 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":282,"threads_started":1}
I20260928 05:17:46.832180 6462 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:17:46.912075 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 6339
I20260928 05:17:46.924348 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 5979
I20260928 05:17:46.925486 6041 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:17:47.023265 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 5979
2026-09-28T05:17:47Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3559 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:17:47 dist-test-slave-whsn krb5kdc[6494](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:17:47 dist-test-slave-whsn krb5kdc[6494](info): set up 2 sockets
Sep 28 05:17:47 dist-test-slave-whsn krb5kdc[6494](info): commencing operation
krb5kdc: starting...
W20260928 05:17:48.005210 388 mini_kdc.cc:121] Time spent starting KDC: real 0.946s user 0.011s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/krb5kdc/test-user.keytab.
Sep 28 05:17:48 dist-test-slave-whsn krb5kdc[6494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572668, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:17:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:17:48Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260928 05:17:48.173573 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38331
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38331
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:17:48.467787 6507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:17:48.468055 6507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:17:48.468153 6507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:17:48.474979 6507 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:17:48.475075 6507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:17:48.475126 6507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:17:48.475179 6507 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:17:48.475220 6507 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:17:48.484381 6507 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38331
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:17:48.485572 6507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:17:48.486713 6507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:17:48.497123 6513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:48.501147 6512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:48.501998 6515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:17:48.503692 6507 server_base.cc:1034] running on GCE node
I20260928 05:17:48.504586 6507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:17:48.506124 6507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:17:48.507314 6507 hybrid_clock.cc:648] HybridClock initialized: now 1790572668507259 us; error 47 us; skew 500 ppm
I20260928 05:17:48.507625 6507 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:17:48 dist-test-slave-whsn krb5kdc[6494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572668, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:17:48.514348 6507 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260928 05:17:48.515630 6507 webserver.cc:460] Webserver started at http://127.0.97.62:40903/ using document root <none> and password file <none>
I20260928 05:17:48.516222 6507 fs_manager.cc:359] Metadata directory not provided
I20260928 05:17:48.516310 6507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:17:48.516543 6507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:17:48.519201 6507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/master-0/data/instance:
uuid: "401619c49c134f59a36d3d960b3589ee"
format_stamp: "Formatted at 2026-09-28 05:17:48 on dist-test-slave-whsn"
server_key: "1273f0e25874f324e99918112edb77d6"
server_key_iv: "e7256f90ed8c320147543274f368b622"
server_key_version: "encryptionkey@0"
I20260928 05:17:48.519846 6507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/master-0/wal/instance:
uuid: "401619c49c134f59a36d3d960b3589ee"
format_stamp: "Formatted at 2026-09-28 05:17:48 on dist-test-slave-whsn"
server_key: "1273f0e25874f324e99918112edb77d6"
server_key_iv: "e7256f90ed8c320147543274f368b622"
server_key_version: "encryptionkey@0"
I20260928 05:17:48.524477 6507 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260928 05:17:48.527567 6522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:48.528766 6507 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 05:17:48.528946 6507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/master-0/wal
uuid: "401619c49c134f59a36d3d960b3589ee"
format_stamp: "Formatted at 2026-09-28 05:17:48 on dist-test-slave-whsn"
server_key: "1273f0e25874f324e99918112edb77d6"
server_key_iv: "e7256f90ed8c320147543274f368b622"
server_key_version: "encryptionkey@0"
I20260928 05:17:48.529131 6507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:17:48.545681 6507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:17:48.549408 6507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:17:48.549655 6507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:17:48.565436 6507 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38331
I20260928 05:17:48.565464 6573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38331 every 8 connection(s)
I20260928 05:17:48.566694 6507 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:17:48.570137 6574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:17:48.572103 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 6507
I20260928 05:17:48.572288 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/master-0/wal/instance
I20260928 05:17:48.572891 388 external_mini_cluster.cc:1428] Setting key 3859dac8725ed90ec3b3323b04f15dfc
I20260928 05:17:48.580917 6574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 401619c49c134f59a36d3d960b3589ee: Bootstrap starting.
I20260928 05:17:48.584299 6574 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 401619c49c134f59a36d3d960b3589ee: Neither blocks nor log segments found. Creating new log.
I20260928 05:17:48.585532 6574 log.cc:824] T 00000000000000000000000000000000 P 401619c49c134f59a36d3d960b3589ee: Log is configured to *not* fsync() on all Append() calls
Sep 28 05:17:48 dist-test-slave-whsn krb5kdc[6494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572668, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260928 05:17:48.588116 6574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 401619c49c134f59a36d3d960b3589ee: No bootstrap required, opened a new log
I20260928 05:17:48.593853 6577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:48.574767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42512 (local address 127.0.97.62:38331)
0928 05:17:48.575501 (+ 734us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:48.575512 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:48.575549 (+ 37us) server_negotiation.cc:407] Connection header received
0928 05:17:48.577139 (+ 1590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:48.577161 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:48.577527 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:48.577835 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:48.579435 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:48.580648 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:48.582282 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:48.582924 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:48.583049 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:48.587428 (+ 4379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:48.587435 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:48.587447 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:48.587482 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:48.590247 (+ 2765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:48.590868 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:48.590876 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:48.590881 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:48.591034 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:48.591360 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:48.591364 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:48.591366 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:48.591849 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:48.592122 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:48.592532 (+ 410us) server_negotiation.cc:299] Negotiation successful
0928 05:17:48.592927 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":341,"threads_started":1}
I20260928 05:17:48.595518 6574 raft_consensus.cc:348] T 00000000000000000000000000000000 P 401619c49c134f59a36d3d960b3589ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "401619c49c134f59a36d3d960b3589ee" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38331 } }
I20260928 05:17:48.595899 6574 raft_consensus.cc:374] T 00000000000000000000000000000000 P 401619c49c134f59a36d3d960b3589ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:17:48.596011 6574 raft_consensus.cc:729] T 00000000000000000000000000000000 P 401619c49c134f59a36d3d960b3589ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 401619c49c134f59a36d3d960b3589ee, State: Initialized, Role: FOLLOWER
I20260928 05:17:48.596558 6574 consensus_queue.cc:260] T 00000000000000000000000000000000 P 401619c49c134f59a36d3d960b3589ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "401619c49c134f59a36d3d960b3589ee" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38331 } }
I20260928 05:17:48.596760 6574 raft_consensus.cc:388] T 00000000000000000000000000000000 P 401619c49c134f59a36d3d960b3589ee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:17:48.596884 6574 raft_consensus.cc:482] T 00000000000000000000000000000000 P 401619c49c134f59a36d3d960b3589ee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:17:48.597018 6574 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 401619c49c134f59a36d3d960b3589ee [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:17:48.598850 6574 raft_consensus.cc:504] T 00000000000000000000000000000000 P 401619c49c134f59a36d3d960b3589ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "401619c49c134f59a36d3d960b3589ee" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38331 } }
I20260928 05:17:48.599243 6574 leader_election.cc:302] T 00000000000000000000000000000000 P 401619c49c134f59a36d3d960b3589ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 401619c49c134f59a36d3d960b3589ee; no voters:
I20260928 05:17:48.599723 6574 leader_election.cc:288] T 00000000000000000000000000000000 P 401619c49c134f59a36d3d960b3589ee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:17:48.599803 6579 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 401619c49c134f59a36d3d960b3589ee [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:17:48.600876 6579 raft_consensus.cc:686] T 00000000000000000000000000000000 P 401619c49c134f59a36d3d960b3589ee [term 1 LEADER]: Becoming Leader. State: Replica: 401619c49c134f59a36d3d960b3589ee, State: Running, Role: LEADER
I20260928 05:17:48.601317 6579 consensus_queue.cc:237] T 00000000000000000000000000000000 P 401619c49c134f59a36d3d960b3589ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "401619c49c134f59a36d3d960b3589ee" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38331 } }
I20260928 05:17:48.601604 6574 sys_catalog.cc:564] T 00000000000000000000000000000000 P 401619c49c134f59a36d3d960b3589ee [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:17:48.603686 6581 sys_catalog.cc:455] T 00000000000000000000000000000000 P 401619c49c134f59a36d3d960b3589ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 401619c49c134f59a36d3d960b3589ee. Latest consensus state: current_term: 1 leader_uuid: "401619c49c134f59a36d3d960b3589ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "401619c49c134f59a36d3d960b3589ee" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38331 } } }
I20260928 05:17:48.603698 6580 sys_catalog.cc:455] T 00000000000000000000000000000000 P 401619c49c134f59a36d3d960b3589ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "401619c49c134f59a36d3d960b3589ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "401619c49c134f59a36d3d960b3589ee" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38331 } } }
I20260928 05:17:48.604065 6580 sys_catalog.cc:458] T 00000000000000000000000000000000 P 401619c49c134f59a36d3d960b3589ee [sys.catalog]: This master's current role is: LEADER
I20260928 05:17:48.604068 6581 sys_catalog.cc:458] T 00000000000000000000000000000000 P 401619c49c134f59a36d3d960b3589ee [sys.catalog]: This master's current role is: LEADER
I20260928 05:17:48.605741 6585 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:17:48.610466 6585 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:17:48.617861 6585 catalog_manager.cc:1347] Generated new cluster ID: 617aa3f5fc464ce4a48eb676675ac7d8
I20260928 05:17:48.617959 6585 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:17:48.641220 6585 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:17:48.642419 6585 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:17:48.651676 6585 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 401619c49c134f59a36d3d960b3589ee: Generated new TSK 0
I20260928 05:17:48.652370 6585 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260928 05:17:48.704046 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38331
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:17:49.009301 6600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:17:49.009568 6600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:17:49.009690 6600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:17:49.016633 6600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:17:49.016724 6600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:17:49.016837 6600 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 05:17:49.025954 6600 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38331
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:17:49.027156 6600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:17:49.028393 6600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:17:49.040303 6605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:49.040275 6608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:49.040333 6606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:17:49.041378 6600 server_base.cc:1034] running on GCE node
I20260928 05:17:49.041903 6600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:17:49.042754 6600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:17:49.043946 6600 hybrid_clock.cc:648] HybridClock initialized: now 1790572669043886 us; error 46 us; skew 500 ppm
I20260928 05:17:49.044224 6600 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:17:49 dist-test-slave-whsn krb5kdc[6494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572669, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:17:49.047147 6600 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260928 05:17:49.048496 6600 webserver.cc:460] Webserver started at http://127.0.97.1:34703/ using document root <none> and password file <none>
I20260928 05:17:49.049167 6600 fs_manager.cc:359] Metadata directory not provided
I20260928 05:17:49.049273 6600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:17:49.049537 6600 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:17:49.052210 6600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/ts-0/data/instance:
uuid: "dafeb19659594f5abdc1fc79acdfb9df"
format_stamp: "Formatted at 2026-09-28 05:17:49 on dist-test-slave-whsn"
server_key: "7a12a4b12aad7e4e1f4ff9c36a343b9d"
server_key_iv: "44848d1908588bee0444a5c3b11c713b"
server_key_version: "encryptionkey@0"
I20260928 05:17:49.053035 6600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/ts-0/wal/instance:
uuid: "dafeb19659594f5abdc1fc79acdfb9df"
format_stamp: "Formatted at 2026-09-28 05:17:49 on dist-test-slave-whsn"
server_key: "7a12a4b12aad7e4e1f4ff9c36a343b9d"
server_key_iv: "44848d1908588bee0444a5c3b11c713b"
server_key_version: "encryptionkey@0"
I20260928 05:17:49.057742 6600 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 05:17:49.061013 6615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:49.062299 6600 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 05:17:49.062449 6600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/ts-0/wal
uuid: "dafeb19659594f5abdc1fc79acdfb9df"
format_stamp: "Formatted at 2026-09-28 05:17:49 on dist-test-slave-whsn"
server_key: "7a12a4b12aad7e4e1f4ff9c36a343b9d"
server_key_iv: "44848d1908588bee0444a5c3b11c713b"
server_key_version: "encryptionkey@0"
I20260928 05:17:49.062633 6600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:17:49.078569 6600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:17:49.081993 6600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:17:49.082233 6600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:17:49.083066 6600 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:17:49.084281 6600 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:17:49.084353 6600 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:49.084461 6600 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:17:49.084514 6600 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:49.116966 6600 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43633
I20260928 05:17:49.117009 6727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43633 every 8 connection(s)
I20260928 05:17:49.118134 6600 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/ts-0/data/info.pb
I20260928 05:17:49.119719 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 6600
I20260928 05:17:49.119923 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/ts-0/wal/instance
I20260928 05:17:49.120677 388 external_mini_cluster.cc:1428] Setting key 50388e9b008754643565d3e9401e11b7
Sep 28 05:17:49 dist-test-slave-whsn krb5kdc[6494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572669, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260928 05:17:49.138491 6730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:49.120528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33883 (local address 127.0.97.62:38331)
0928 05:17:49.120941 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:49.120947 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:49.122269 (+ 1322us) server_negotiation.cc:407] Connection header received
0928 05:17:49.124128 (+ 1859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:49.124134 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:49.124208 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:49.124313 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:49.125796 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:49.126754 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:49.128199 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:49.128576 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:49.128679 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:49.132327 (+ 3648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:49.132335 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:49.132339 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:49.132390 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:49.135554 (+ 3164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:49.136203 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:49.136209 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:49.136213 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:49.136331 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:49.136719 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:49.136725 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:49.136729 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:49.137069 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:49.137200 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:49.137941 (+ 741us) server_negotiation.cc:299] Negotiation successful
0928 05:17:49.138152 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":197,"threads_started":1}
I20260928 05:17:49.139400 6731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:49.121244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38331 (local address 127.0.97.1:33883)
0928 05:17:49.122105 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:49.122155 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:49.123861 (+ 1706us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:49.124550 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:49.124558 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:49.125104 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:17:49.125643 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:49.125660 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:49.126877 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:49.126880 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:49.128065 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:49.128076 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:49.128774 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:49.128783 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:49.129086 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:49.129713 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:17:49.129739 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:17:49.132099 (+ 2360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:17:49.135690 (+ 3591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:49.135696 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:49.135707 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:49.136080 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:49.136435 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:49.136438 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:49.136441 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:49.136610 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:49.137202 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:17:49.137209 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:17:49.137592 (+ 383us) client_negotiation.cc:769] Sending connection context
0928 05:17:49.137829 (+ 237us) client_negotiation.cc:240] Negotiation successful
0928 05:17:49.138293 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":311,"threads_started":1}
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260928 05:17:49.140875 6728 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38331
I20260928 05:17:49.141170 6728 heartbeater.cc:464] Registering TS with master...
I20260928 05:17:49.141835 6728 heartbeater.cc:511] Master 127.0.97.62:38331 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
I20260928 05:17:49.143875 6539 ts_manager.cc:194] Registered new tserver with Master: dafeb19659594f5abdc1fc79acdfb9df (127.0.97.1:43633)
I20260928 05:17:49.145342 6539 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:33883
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260928 05:17:49.162212 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38331
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:17:49.456471 6734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:17:49.456720 6734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:17:49.456867 6734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:17:49.463752 6734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:17:49.463843 6734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:17:49.463992 6734 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 05:17:49.473173 6734 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38331
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:17:49.474330 6734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:17:49.475519 6734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:17:49.487427 6742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:49.487528 6739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:49.487464 6740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:17:49.488557 6734 server_base.cc:1034] running on GCE node
I20260928 05:17:49.489172 6734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:17:49.490069 6734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:17:49.491254 6734 hybrid_clock.cc:648] HybridClock initialized: now 1790572669491207 us; error 38 us; skew 500 ppm
I20260928 05:17:49.491547 6734 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:17:49 dist-test-slave-whsn krb5kdc[6494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572669, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:17:49.494454 6734 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260928 05:17:49.495759 6734 webserver.cc:460] Webserver started at http://127.0.97.2:45031/ using document root <none> and password file <none>
I20260928 05:17:49.496366 6734 fs_manager.cc:359] Metadata directory not provided
I20260928 05:17:49.496482 6734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:17:49.496747 6734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:17:49.499504 6734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/ts-1/data/instance:
uuid: "176eae27f2854c7a84d8a0e5c342d0a9"
format_stamp: "Formatted at 2026-09-28 05:17:49 on dist-test-slave-whsn"
server_key: "39e5977ad88887f426576145a904f05c"
server_key_iv: "d8e71ec1cd5a98733d0e1dda27b80903"
server_key_version: "encryptionkey@0"
I20260928 05:17:49.500195 6734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/ts-1/wal/instance:
uuid: "176eae27f2854c7a84d8a0e5c342d0a9"
format_stamp: "Formatted at 2026-09-28 05:17:49 on dist-test-slave-whsn"
server_key: "39e5977ad88887f426576145a904f05c"
server_key_iv: "d8e71ec1cd5a98733d0e1dda27b80903"
server_key_version: "encryptionkey@0"
I20260928 05:17:49.505012 6734 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260928 05:17:49.508082 6749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:49.509323 6734 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 05:17:49.509470 6734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/ts-1/wal
uuid: "176eae27f2854c7a84d8a0e5c342d0a9"
format_stamp: "Formatted at 2026-09-28 05:17:49 on dist-test-slave-whsn"
server_key: "39e5977ad88887f426576145a904f05c"
server_key_iv: "d8e71ec1cd5a98733d0e1dda27b80903"
server_key_version: "encryptionkey@0"
I20260928 05:17:49.509655 6734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:17:49.525200 6734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:17:49.528546 6734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:17:49.528779 6734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:17:49.529641 6734 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:17:49.530804 6734 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:17:49.530876 6734 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:49.530992 6734 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:17:49.531057 6734 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:49.559796 6734 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33089
I20260928 05:17:49.559854 6861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33089 every 8 connection(s)
I20260928 05:17:49.561110 6734 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/ts-1/data/info.pb
I20260928 05:17:49.566831 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 6734
I20260928 05:17:49.567039 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/ts-1/wal/instance
I20260928 05:17:49.567551 388 external_mini_cluster.cc:1428] Setting key 13cfbd50f2a2adde0c7d4b6f832eda76
Sep 28 05:17:49 dist-test-slave-whsn krb5kdc[6494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572669, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260928 05:17:49.585129 6730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:49.563358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43045 (local address 127.0.97.62:38331)
0928 05:17:49.563550 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:49.563554 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:49.564639 (+ 1085us) server_negotiation.cc:407] Connection header received
0928 05:17:49.567063 (+ 2424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:49.567069 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:49.567172 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:49.567289 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:49.569077 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:49.569825 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:49.571269 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:49.571688 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:49.571789 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:49.578881 (+ 7092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:49.578887 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:49.578891 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:49.578926 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:49.582022 (+ 3096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:49.582844 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:49.582850 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:49.582854 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:49.582996 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:49.583368 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:49.583372 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:49.583375 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:49.583688 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:49.583823 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:49.584568 (+ 745us) server_negotiation.cc:299] Negotiation successful
0928 05:17:49.584794 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 05:17:49.585892 6864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:49.563813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38331 (local address 127.0.97.2:43045)
0928 05:17:49.564483 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:49.564520 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:49.566741 (+ 2221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:49.567508 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:49.567519 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:49.568140 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:17:49.568879 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:49.568902 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:49.569966 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:49.569970 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:49.571140 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:49.571152 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:49.571879 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:49.571887 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:49.572098 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:49.575946 (+ 3848us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:17:49.575968 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:17:49.578649 (+ 2681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:17:49.582158 (+ 3509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:49.582167 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:49.582183 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:49.582710 (+ 527us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:49.583090 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:49.583093 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:49.583096 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:49.583267 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:49.583854 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:17:49.583861 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:17:49.584209 (+ 348us) client_negotiation.cc:769] Sending connection context
0928 05:17:49.584426 (+ 217us) client_negotiation.cc:240] Negotiation successful
0928 05:17:49.584796 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":253,"threads_started":1}
I20260928 05:17:49.587144 6862 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38331
I20260928 05:17:49.587395 6862 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260928 05:17:49.588043 6862 heartbeater.cc:511] Master 127.0.97.62:38331 requested a full tablet report, sending...
I20260928 05:17:49.589339 6539 ts_manager.cc:194] Registered new tserver with Master: 176eae27f2854c7a84d8a0e5c342d0a9 (127.0.97.2:33089)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
I20260928 05:17:49.590278 6539 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:43045
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260928 05:17:49.609472 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38331
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:17:49.920229 6867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:17:49.920485 6867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:17:49.920596 6867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:17:49.927462 6867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:17:49.927567 6867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:17:49.927713 6867 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 05:17:49.936794 6867 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38331
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:17:49.937989 6867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:17:49.939203 6867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:17:49.951040 6872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:49.951005 6873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:49.951069 6875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:17:49.951756 6867 server_base.cc:1034] running on GCE node
I20260928 05:17:49.952297 6867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:17:49.953161 6867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:17:49.954353 6867 hybrid_clock.cc:648] HybridClock initialized: now 1790572669954319 us; error 48 us; skew 500 ppm
I20260928 05:17:49.954624 6867 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:17:49 dist-test-slave-whsn krb5kdc[6494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572669, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:17:49.957372 6867 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260928 05:17:49.958680 6867 webserver.cc:460] Webserver started at http://127.0.97.3:39949/ using document root <none> and password file <none>
I20260928 05:17:49.959306 6867 fs_manager.cc:359] Metadata directory not provided
I20260928 05:17:49.959417 6867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:17:49.959672 6867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:17:49.962319 6867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/ts-2/data/instance:
uuid: "a2d63277ac00458a83c7cb684be08738"
format_stamp: "Formatted at 2026-09-28 05:17:49 on dist-test-slave-whsn"
server_key: "e0fd083962b0f78af493447106d4353c"
server_key_iv: "7b8d610014d92e5e40b85449ecd4a27c"
server_key_version: "encryptionkey@0"
I20260928 05:17:49.963016 6867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/ts-2/wal/instance:
uuid: "a2d63277ac00458a83c7cb684be08738"
format_stamp: "Formatted at 2026-09-28 05:17:49 on dist-test-slave-whsn"
server_key: "e0fd083962b0f78af493447106d4353c"
server_key_iv: "7b8d610014d92e5e40b85449ecd4a27c"
server_key_version: "encryptionkey@0"
I20260928 05:17:49.967888 6867 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 05:17:49.971024 6882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:49.972203 6867 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260928 05:17:49.972334 6867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/ts-2/wal
uuid: "a2d63277ac00458a83c7cb684be08738"
format_stamp: "Formatted at 2026-09-28 05:17:49 on dist-test-slave-whsn"
server_key: "e0fd083962b0f78af493447106d4353c"
server_key_iv: "7b8d610014d92e5e40b85449ecd4a27c"
server_key_version: "encryptionkey@0"
I20260928 05:17:49.972496 6867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:17:50.006958 6867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:17:50.010740 6867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:17:50.010993 6867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:17:50.011864 6867 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:17:50.013155 6867 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:17:50.013229 6867 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:50.013347 6867 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:17:50.013396 6867 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:50.041383 6867 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46223
I20260928 05:17:50.041426 6994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46223 every 8 connection(s)
I20260928 05:17:50.042537 6867 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/ts-2/data/info.pb
I20260928 05:17:50.044745 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 6867
I20260928 05:17:50.044953 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/ts-2/wal/instance
I20260928 05:17:50.045442 388 external_mini_cluster.cc:1428] Setting key cad72213489adda0deb96e5b2cfe1f16
Sep 28 05:17:50 dist-test-slave-whsn krb5kdc[6494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572669, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260928 05:17:50.061124 6730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:50.044815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51971 (local address 127.0.97.62:38331)
0928 05:17:50.045040 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:50.045047 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:50.046414 (+ 1367us) server_negotiation.cc:407] Connection header received
0928 05:17:50.048479 (+ 2065us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:50.048485 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:50.048559 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:50.048639 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:50.050110 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:50.050806 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:50.052313 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:50.052702 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:50.052804 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:50.056149 (+ 3345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:50.056155 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:50.056158 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:50.056187 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:50.058420 (+ 2233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:50.059109 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:50.059114 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:50.059118 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:50.059254 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:50.059647 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:50.059651 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:50.059653 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:50.059981 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:50.060119 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:50.060713 (+ 594us) server_negotiation.cc:299] Negotiation successful
0928 05:17:50.060895 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260928 05:17:50.061928 6997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:50.045483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38331 (local address 127.0.97.3:51971)
0928 05:17:50.046266 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:50.046309 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:50.048242 (+ 1933us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:50.048910 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:50.048919 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:50.049396 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:17:50.049957 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:50.049975 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:50.050951 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:50.050954 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:50.052184 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:50.052195 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:50.052939 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:50.052947 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:50.053244 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:50.053787 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:17:50.053812 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:17:50.055960 (+ 2148us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:17:50.058550 (+ 2590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:50.058557 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:50.058571 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:50.058991 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:50.059351 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:50.059356 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:50.059360 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:50.059531 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:50.060112 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:17:50.060120 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:17:50.060400 (+ 280us) client_negotiation.cc:769] Sending connection context
0928 05:17:50.060608 (+ 208us) client_negotiation.cc:240] Negotiation successful
0928 05:17:50.060993 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":308,"threads_started":1}
I20260928 05:17:50.063105 6995 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38331
I20260928 05:17:50.063351 6995 heartbeater.cc:464] Registering TS with master...
I20260928 05:17:50.064028 6995 heartbeater.cc:511] Master 127.0.97.62:38331 requested a full tablet report, sending...
I20260928 05:17:50.065294 6539 ts_manager.cc:194] Registered new tserver with Master: a2d63277ac00458a83c7cb684be08738 (127.0.97.3:46223)
I20260928 05:17:50.066088 6539 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:51971
I20260928 05:17:50.072067 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/krb5kdc/oryx_127.0.97.61.keytab.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/krb5kdc/oryx_127.0.97.61.keytab.
I20260928 05:17:50.112617 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:40533
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38331,127.0.97.61:40533
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 05:17:50.148376 6728 heartbeater.cc:503] Master 127.0.97.62:38331 was elected leader, sending a full tablet report...
W20260928 05:17:50.404124 7000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:17:50.404363 7000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:17:50.404488 7000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:17:50.411567 7000 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:17:50.411700 7000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:17:50.411793 7000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:17:50.411882 7000 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:17:50.411948 7000 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:17:50.421802 7000 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38331,127.0.97.61:40533
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:40533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
W20260928 05:17:50.422155 7000 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.97.62:38331,127.0.97.61:40533'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260928 05:17:50.423141 7000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:17:50.424449 7000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:17:50.435202 7006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:50.435266 7005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:50.435771 7008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:17:50.435859 7000 server_base.cc:1034] running on GCE node
I20260928 05:17:50.436587 7000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:17:50.437914 7000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:17:50.439110 7000 hybrid_clock.cc:648] HybridClock initialized: now 1790572670439052 us; error 42 us; skew 500 ppm
I20260928 05:17:50.439399 7000 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:17:50 dist-test-slave-whsn krb5kdc[6494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572670, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:17:50.442230 7000 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260928 05:17:50.443485 7000 webserver.cc:460] Webserver started at http://127.0.97.61:35471/ using document root <none> and password file <none>
I20260928 05:17:50.444123 7000 fs_manager.cc:359] Metadata directory not provided
I20260928 05:17:50.444216 7000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:17:50.444491 7000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:17:50.447268 7000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/master-1/data/instance:
uuid: "109bdf09ab804bf1b13ca35cce2e6bcf"
format_stamp: "Formatted at 2026-09-28 05:17:50 on dist-test-slave-whsn"
server_key: "00ed56967467c1396881c08100e218dd"
server_key_iv: "bc982ab0c79e989cae5d765bdb9b90d5"
server_key_version: "encryptionkey@0"
I20260928 05:17:50.447952 7000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/master-1/wal/instance:
uuid: "109bdf09ab804bf1b13ca35cce2e6bcf"
format_stamp: "Formatted at 2026-09-28 05:17:50 on dist-test-slave-whsn"
server_key: "00ed56967467c1396881c08100e218dd"
server_key_iv: "bc982ab0c79e989cae5d765bdb9b90d5"
server_key_version: "encryptionkey@0"
I20260928 05:17:50.452567 7000 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260928 05:17:50.455644 7015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:50.456928 7000 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260928 05:17:50.457079 7000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/master-1/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/master-1/wal
uuid: "109bdf09ab804bf1b13ca35cce2e6bcf"
format_stamp: "Formatted at 2026-09-28 05:17:50 on dist-test-slave-whsn"
server_key: "00ed56967467c1396881c08100e218dd"
server_key_iv: "bc982ab0c79e989cae5d765bdb9b90d5"
server_key_version: "encryptionkey@0"
I20260928 05:17:50.457252 7000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:17:50.491979 7000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:17:50.496887 7000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:17:50.497176 7000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:17:50.513810 7000 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:40533
I20260928 05:17:50.513829 7066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:40533 every 8 connection(s)
I20260928 05:17:50.515061 7000 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/master-1/data/info.pb
I20260928 05:17:50.517210 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 7000
I20260928 05:17:50.518931 7067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:17:50.525467 7067 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38331 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 05:17:50 dist-test-slave-whsn krb5kdc[6494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572668, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Sep 28 05:17:50 dist-test-slave-whsn krb5kdc[6494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572670, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260928 05:17:50.542809 6730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:50.528574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42528 (local address 127.0.97.62:38331)
0928 05:17:50.528757 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:50.528763 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:50.529392 (+ 629us) server_negotiation.cc:407] Connection header received
0928 05:17:50.529580 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:50.529585 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:50.529644 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:50.529723 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:50.530935 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:50.531983 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:50.533454 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:50.534048 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:50.534180 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:50.537192 (+ 3012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:50.537198 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:50.537202 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:50.537244 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:50.539961 (+ 2717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:50.540706 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:50.540711 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:50.540714 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:50.540885 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:50.541275 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:50.541280 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:50.541283 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:50.541645 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:50.541787 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:50.542352 (+ 565us) server_negotiation.cc:299] Negotiation successful
0928 05:17:50.542556 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 05:17:50.543747 7070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:50.519416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39574 (local address 127.0.97.61:40533)
0928 05:17:50.520250 (+ 834us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:50.520266 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:50.524506 (+ 4240us) server_negotiation.cc:407] Connection header received
0928 05:17:50.526160 (+ 1654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:50.526181 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:50.526564 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:50.526888 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:50.528509 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:50.530399 (+ 1890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:50.531886 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:50.532543 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:50.532709 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:50.536793 (+ 4084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:50.536801 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:50.536811 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:50.536846 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:50.539938 (+ 3092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:50.540717 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:50.540725 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:50.540727 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:50.540905 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:50.541265 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:50.541269 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:50.541272 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:50.541849 (+ 577us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:50.542038 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:50.542439 (+ 401us) server_negotiation.cc:299] Negotiation successful
0928 05:17:50.542836 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":151,"server-negotiator.queue_time_us":602,"thread_start_us":351,"threads_started":1}
I20260928 05:17:50.543789 7071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:50.528564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38331 (local address 127.0.0.1:42528)
0928 05:17:50.529259 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:50.529302 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:50.529445 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:50.529810 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:50.529820 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:50.530564 (+ 744us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:17:50.530785 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:50.530793 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:50.532110 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:50.532116 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:50.533328 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:50.533342 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:50.534235 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:50.534240 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:50.534350 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:50.534841 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:17:50.534868 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:17:50.537015 (+ 2147us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:17:50.540086 (+ 3071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:50.540096 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:50.540107 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:50.540597 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:50.540986 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:50.540991 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:50.540995 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:50.541169 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:50.541772 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:17:50.541777 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:17:50.541983 (+ 206us) client_negotiation.cc:769] Sending connection context
0928 05:17:50.542251 (+ 268us) client_negotiation.cc:240] Negotiation successful
0928 05:17:50.542698 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":336,"threads_started":1}
I20260928 05:17:50.545552 7067 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40533 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 05:17:50 dist-test-slave-whsn krb5kdc[6494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572670, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260928 05:17:50.555626 7071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:50.546087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40533 (local address 127.0.0.1:39584)
0928 05:17:50.546249 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:50.546266 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:50.546353 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:50.546626 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:50.546629 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:50.546903 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:17:50.547013 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:50.547019 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:50.547844 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:50.547848 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:50.548647 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:50.548656 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:50.549358 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:50.549362 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:50.549498 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:50.549874 (+ 376us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:17:50.549889 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:17:50.551569 (+ 1680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:17:50.553831 (+ 2262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:50.553834 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:50.553837 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:50.554173 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:50.554462 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:50.554465 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:50.554467 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:50.554573 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:50.555000 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:17:50.555003 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:17:50.555106 (+ 103us) client_negotiation.cc:769] Sending connection context
0928 05:17:50.555217 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 05:17:50.555388 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260928 05:17:50.555763 7070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:50.546230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39584 (local address 127.0.97.61:40533)
0928 05:17:50.546359 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:50.546364 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:50.546379 (+ 15us) server_negotiation.cc:407] Connection header received
0928 05:17:50.546444 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:50.546447 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:50.546509 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:50.546589 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:50.547118 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:50.547750 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:50.548746 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:50.549250 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:50.549340 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:50.551717 (+ 2377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:50.551722 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:50.551725 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:50.551750 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:50.553730 (+ 1980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:50.554258 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:50.554261 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:50.554263 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:50.554371 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:50.554662 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:50.554665 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:50.554667 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:50.554905 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:50.555015 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:50.555286 (+ 271us) server_negotiation.cc:299] Negotiation successful
0928 05:17:50.555492 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":41}
I20260928 05:17:50.559957 7067 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 109bdf09ab804bf1b13ca35cce2e6bcf: Bootstrap starting.
I20260928 05:17:50.562871 7067 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 109bdf09ab804bf1b13ca35cce2e6bcf: Neither blocks nor log segments found. Creating new log.
I20260928 05:17:50.563783 7067 log.cc:824] T 00000000000000000000000000000000 P 109bdf09ab804bf1b13ca35cce2e6bcf: Log is configured to *not* fsync() on all Append() calls
I20260928 05:17:50.566120 7067 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 109bdf09ab804bf1b13ca35cce2e6bcf: No bootstrap required, opened a new log
I20260928 05:17:50.572536 7067 raft_consensus.cc:348] T 00000000000000000000000000000000 P 109bdf09ab804bf1b13ca35cce2e6bcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401619c49c134f59a36d3d960b3589ee" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38331 } } peers { permanent_uuid: "109bdf09ab804bf1b13ca35cce2e6bcf" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40533 } }
I20260928 05:17:50.572813 7067 raft_consensus.cc:374] T 00000000000000000000000000000000 P 109bdf09ab804bf1b13ca35cce2e6bcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:17:50.572919 7067 raft_consensus.cc:729] T 00000000000000000000000000000000 P 109bdf09ab804bf1b13ca35cce2e6bcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 109bdf09ab804bf1b13ca35cce2e6bcf, State: Initialized, Role: FOLLOWER
I20260928 05:17:50.573407 7067 consensus_queue.cc:260] T 00000000000000000000000000000000 P 109bdf09ab804bf1b13ca35cce2e6bcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401619c49c134f59a36d3d960b3589ee" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38331 } } peers { permanent_uuid: "109bdf09ab804bf1b13ca35cce2e6bcf" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40533 } }
I20260928 05:17:50.574128 7073 sys_catalog.cc:455] T 00000000000000000000000000000000 P 109bdf09ab804bf1b13ca35cce2e6bcf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401619c49c134f59a36d3d960b3589ee" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38331 } } peers { permanent_uuid: "109bdf09ab804bf1b13ca35cce2e6bcf" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40533 } } }
I20260928 05:17:50.574398 7073 sys_catalog.cc:458] T 00000000000000000000000000000000 P 109bdf09ab804bf1b13ca35cce2e6bcf [sys.catalog]: This master's current role is: FOLLOWER
I20260928 05:17:50.574934 7067 sys_catalog.cc:564] T 00000000000000000000000000000000 P 109bdf09ab804bf1b13ca35cce2e6bcf [sys.catalog]: configured and running, proceeding with master startup.
W20260928 05:17:50.583559 7084 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 109bdf09ab804bf1b13ca35cce2e6bcf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 05:17:50.583695 7084 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 05:17:50.584834 7000 master_runner.cc:428] Detected that this master 109bdf09ab804bf1b13ca35cce2e6bcf is joining an existing cluster
I20260928 05:17:50.584975 7000 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.61:40533
I20260928 05:17:50.592692 6862 heartbeater.cc:503] Master 127.0.97.62:38331 was elected leader, sending a full tablet report...
I20260928 05:17:50.608903 6730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:50.591208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42540 (local address 127.0.97.62:38331)
0928 05:17:50.591675 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:50.591681 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:50.594868 (+ 3187us) server_negotiation.cc:407] Connection header received
0928 05:17:50.595190 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:50.595196 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:50.595271 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:50.595355 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:50.596611 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:50.597704 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:50.599189 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:50.599811 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:50.599922 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:50.602391 (+ 2469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:50.602396 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:50.602401 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:50.602430 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:50.605664 (+ 3234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:50.606420 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:50.606426 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:50.606429 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:50.606586 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:50.607271 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:50.607276 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:50.607280 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:50.607673 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:50.607822 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:50.608300 (+ 478us) server_negotiation.cc:299] Negotiation successful
0928 05:17:50.608533 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344}
I20260928 05:17:50.609596 7090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:50.591345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38331 (local address 127.0.0.1:42540)
0928 05:17:50.594769 (+ 3424us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:50.594792 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:50.595042 (+ 250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:50.595385 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:50.595390 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:50.596039 (+ 649us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:17:50.596467 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:50.596480 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:50.597858 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:50.597864 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:50.599062 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:50.599073 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:50.600142 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:50.600148 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:50.600387 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:50.601111 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:17:50.601134 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:17:50.602244 (+ 1110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:17:50.605780 (+ 3536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:50.605785 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:50.605789 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:50.606284 (+ 495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:50.606957 (+ 673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:50.606962 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:50.606966 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:50.607141 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:50.607805 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:17:50.607810 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:17:50.608014 (+ 204us) client_negotiation.cc:769] Sending connection context
0928 05:17:50.608179 (+ 165us) client_negotiation.cc:240] Negotiation successful
0928 05:17:50.608449 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3283,"mutex_wait_us":23,"thread_start_us":3157,"threads_started":1}
I20260928 05:17:50.611589 7070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:50.591726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39600 (local address 127.0.97.61:40533)
0928 05:17:50.591892 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:50.591898 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:50.595295 (+ 3397us) server_negotiation.cc:407] Connection header received
0928 05:17:50.595434 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:50.595439 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:50.595502 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:50.595552 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:50.596608 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:50.597712 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:50.600717 (+ 3005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:50.601346 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:50.601459 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:50.604090 (+ 2631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:50.604096 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:50.604101 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:50.604127 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:50.608083 (+ 3956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:50.609535 (+ 1452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:50.609541 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:50.609544 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:50.609689 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:50.610100 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:50.610104 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:50.610108 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:50.610418 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:50.610553 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:50.611050 (+ 497us) server_negotiation.cc:299] Negotiation successful
0928 05:17:50.611253 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 05:17:50.611665 7073 raft_consensus.cc:482] T 00000000000000000000000000000000 P 109bdf09ab804bf1b13ca35cce2e6bcf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:17:50.611934 7073 raft_consensus.cc:504] T 00000000000000000000000000000000 P 109bdf09ab804bf1b13ca35cce2e6bcf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401619c49c134f59a36d3d960b3589ee" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38331 } } peers { permanent_uuid: "109bdf09ab804bf1b13ca35cce2e6bcf" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40533 } }
I20260928 05:17:50.612602 7091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:50.591405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40533 (local address 127.0.0.1:39600)
0928 05:17:50.595193 (+ 3788us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:50.595216 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:50.595332 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:50.595678 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:50.595681 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:50.596151 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:17:50.596485 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:50.596496 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:50.597902 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:50.597910 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:50.600559 (+ 2649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:50.600571 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:50.601751 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:50.601758 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:50.601930 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:50.602996 (+ 1066us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:17:50.603016 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:17:50.603951 (+ 935us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:17:50.608933 (+ 4982us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:50.608937 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:50.608941 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:50.609410 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:50.609810 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:50.609814 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:50.609818 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:50.609981 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:50.610641 (+ 660us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:17:50.610645 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:17:50.610792 (+ 147us) client_negotiation.cc:769] Sending connection context
0928 05:17:50.611800 (+ 1008us) client_negotiation.cc:240] Negotiation successful
0928 05:17:50.612113 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3673,"mutex_wait_us":42,"thread_start_us":3465,"threads_started":1}
I20260928 05:17:50.613389 7073 leader_election.cc:288] T 00000000000000000000000000000000 P 109bdf09ab804bf1b13ca35cce2e6bcf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 401619c49c134f59a36d3d960b3589ee (127.0.97.62:38331)
I20260928 05:17:50.614168 6549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "109bdf09ab804bf1b13ca35cce2e6bcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "401619c49c134f59a36d3d960b3589ee" is_pre_election: true
I20260928 05:17:50.615137 7016 leader_election.cc:302] T 00000000000000000000000000000000 P 109bdf09ab804bf1b13ca35cce2e6bcf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 109bdf09ab804bf1b13ca35cce2e6bcf; no voters: 401619c49c134f59a36d3d960b3589ee
I20260928 05:17:50.615526 7073 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 109bdf09ab804bf1b13ca35cce2e6bcf [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
WARNING: no policy specified for oryx/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.60@KRBTEST.COM" created.
Sep 28 05:17:50 dist-test-slave-whsn krb5kdc[6494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572668, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260928 05:17:50.637048 7093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:50.616769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40533 (local address 127.0.0.1:39608)
0928 05:17:50.620760 (+ 3991us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:50.620813 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:50.621507 (+ 694us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:50.621979 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:50.621987 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:50.622749 (+ 762us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:17:50.623017 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:50.623030 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:50.624387 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:50.624393 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:50.626000 (+ 1607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:50.626019 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:50.626851 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:50.626858 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:50.627067 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:50.627691 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:17:50.627749 (+ 58us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:17:50.630467 (+ 2718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:17:50.634330 (+ 3863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:50.634341 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:50.634349 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:50.634846 (+ 497us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:50.635254 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:50.635260 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:50.635263 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:50.635413 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:50.635982 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:17:50.635989 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:17:50.636166 (+ 177us) client_negotiation.cc:769] Sending connection context
0928 05:17:50.636337 (+ 171us) client_negotiation.cc:240] Negotiation successful
0928 05:17:50.636638 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3775,"thread_start_us":3633,"threads_started":1}
I20260928 05:17:50.637076 7070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:50.616788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39608 (local address 127.0.97.61:40533)
0928 05:17:50.616970 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:50.616975 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:50.621421 (+ 4446us) server_negotiation.cc:407] Connection header received
0928 05:17:50.621714 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:50.621719 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:50.621817 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:50.621912 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:50.623174 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:50.624254 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:50.626144 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:50.626658 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:50.626754 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:50.630647 (+ 3893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:50.630653 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:50.630668 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:50.630704 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:50.634180 (+ 3476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:50.634982 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:50.634988 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:50.634991 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:50.635145 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:50.635523 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:50.635527 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:50.635531 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:50.635853 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:50.636003 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:50.636484 (+ 481us) server_negotiation.cc:299] Negotiation successful
0928 05:17:50.636696 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 05:17:50.639287 6539 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.61:40533: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "109bdf09ab804bf1b13ca35cce2e6bcf"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.61"
port: 40533
}
attrs {
promote: true
}
}
dest_uuid: "401619c49c134f59a36d3d960b3589ee"
cas_config_opid_index: -1
I20260928 05:17:50.640352 6539 consensus_queue.cc:237] T 00000000000000000000000000000000 P 401619c49c134f59a36d3d960b3589ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "401619c49c134f59a36d3d960b3589ee" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38331 } } peers { permanent_uuid: "109bdf09ab804bf1b13ca35cce2e6bcf" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 40533 } attrs { promote: true } }
I20260928 05:17:50.647081 7096 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 401619c49c134f59a36d3d960b3589ee [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 109bdf09ab804bf1b13ca35cce2e6bcf (127.0.97.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "401619c49c134f59a36d3d960b3589ee" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38331 } } peers { permanent_uuid: "109bdf09ab804bf1b13ca35cce2e6bcf" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 40533 } attrs { promote: true } } }
I20260928 05:17:50.648088 7042 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 109bdf09ab804bf1b13ca35cce2e6bcf [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:17:50.651077 7096 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.61:40533
I20260928 05:17:50.651708 7000 master.cc:537] Master@127.0.97.61:40533 shutting down...
I20260928 05:17:50.651556 7097 sys_catalog.cc:455] T 00000000000000000000000000000000 P 401619c49c134f59a36d3d960b3589ee [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "401619c49c134f59a36d3d960b3589ee" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "401619c49c134f59a36d3d960b3589ee" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38331 } } peers { permanent_uuid: "109bdf09ab804bf1b13ca35cce2e6bcf" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 40533 } attrs { promote: true } } }
I20260928 05:17:50.651799 7097 sys_catalog.cc:458] T 00000000000000000000000000000000 P 401619c49c134f59a36d3d960b3589ee [sys.catalog]: This master's current role is: LEADER
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/krb5kdc/oryx_127.0.97.60.keytab.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/krb5kdc/oryx_127.0.97.60.keytab.
I20260928 05:17:50.653115 7042 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 109bdf09ab804bf1b13ca35cce2e6bcf [term 1 FOLLOWER]: Refusing update from remote peer 401619c49c134f59a36d3d960b3589ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260928 05:17:50.654404 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:39701
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38331,127.0.97.61:40533,127.0.97.60:39701
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 05:17:50.654836 7097 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 401619c49c134f59a36d3d960b3589ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "109bdf09ab804bf1b13ca35cce2e6bcf" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 40533 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.001s
W20260928 05:17:50.657582 6524 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260928 05:17:50.662756 6524 consensus_peers.cc:487] T 00000000000000000000000000000000 P 401619c49c134f59a36d3d960b3589ee -> Peer 109bdf09ab804bf1b13ca35cce2e6bcf (127.0.97.61:40533): Couldn't send request to peer 109bdf09ab804bf1b13ca35cce2e6bcf. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 05:17:50.664549 7000 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 109bdf09ab804bf1b13ca35cce2e6bcf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:17:50.664971 7000 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 109bdf09ab804bf1b13ca35cce2e6bcf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:17:50.665170 7000 tablet_replica.cc:330] T 00000000000000000000000000000000 P 109bdf09ab804bf1b13ca35cce2e6bcf: stopping tablet replica
I20260928 05:17:50.681819 7000 master.cc:552] Master@127.0.97.61:40533 shutdown complete.
I20260928 05:17:50.681902 7000 master_runner.cc:305] Clearing existing system tablet
I20260928 05:17:50.682318 7000 fs_manager.cc:359] Metadata directory not provided
I20260928 05:17:50.682403 7000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:17:50.685214 7000 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.001s
I20260928 05:17:50.689204 7102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:50.689493 7000 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:17:50.689759 7000 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260928 05:17:50.689838 7000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/master-1/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/master-1/wal
uuid: "109bdf09ab804bf1b13ca35cce2e6bcf"
format_stamp: "Formatted at 2026-09-28 05:17:50 on dist-test-slave-whsn"
server_key: "00ed56967467c1396881c08100e218dd"
server_key_iv: "bc982ab0c79e989cae5d765bdb9b90d5"
server_key_version: "encryptionkey@0"
I20260928 05:17:50.694540 7000 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 109bdf09ab804bf1b13ca35cce2e6bcf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:17:50.701051 7000 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 109bdf09ab804bf1b13ca35cce2e6bcf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 05:17:50.701206 7000 log.cc:1196] T 00000000000000000000000000000000 P 109bdf09ab804bf1b13ca35cce2e6bcf: Deleting WAL directory at /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260928 05:17:50.701627 7000 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 109bdf09ab804bf1b13ca35cce2e6bcf: Deleting consensus metadata
I20260928 05:17:50.701867 7000 master_runner.cc:315] Copying system tablet from 127.0.97.62:38331
I20260928 05:17:50.704351 7000 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 109bdf09ab804bf1b13ca35cce2e6bcf: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:38331
I20260928 05:17:50.722726 6730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:50.705201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42556 (local address 127.0.97.62:38331)
0928 05:17:50.705382 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:50.705387 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:50.709376 (+ 3989us) server_negotiation.cc:407] Connection header received
0928 05:17:50.709569 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:50.709574 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:50.709668 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:50.709774 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:50.710963 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:50.712050 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:50.713907 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:50.714538 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:50.714666 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:50.716718 (+ 2052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:50.716725 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:50.716729 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:50.716766 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:50.719965 (+ 3199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:50.720675 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:50.720681 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:50.720685 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:50.720827 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:50.721235 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:50.721240 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:50.721243 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:50.721625 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:50.721764 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:50.722202 (+ 438us) server_negotiation.cc:299] Negotiation successful
0928 05:17:50.722419 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 05:17:50.722795 7108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:50.705181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38331 (local address 127.0.0.1:42556)
0928 05:17:50.709251 (+ 4070us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:50.709275 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:50.709423 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:50.709939 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:50.709944 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:50.710466 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:17:50.710791 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:50.710807 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:50.712260 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:50.712266 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:50.713752 (+ 1486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:50.713767 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:50.714688 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:50.714694 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:50.714867 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:50.715525 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:17:50.715547 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:17:50.716525 (+ 978us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:17:50.720101 (+ 3576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:50.720106 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:50.720111 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:50.720559 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:50.720956 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:50.720961 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:50.720965 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:50.721128 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:50.721741 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:17:50.721749 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:17:50.721909 (+ 160us) client_negotiation.cc:769] Sending connection context
0928 05:17:50.722085 (+ 176us) client_negotiation.cc:240] Negotiation successful
0928 05:17:50.722353 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3924,"thread_start_us":3805,"threads_started":1}
I20260928 05:17:50.723869 6559 tablet_copy_service.cc:140] P 401619c49c134f59a36d3d960b3589ee: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 109bdf09ab804bf1b13ca35cce2e6bcf ({username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:42556)
I20260928 05:17:50.724123 6559 tablet_copy_service.cc:161] P 401619c49c134f59a36d3d960b3589ee: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 109bdf09ab804bf1b13ca35cce2e6bcf at {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:42556: session id = 109bdf09ab804bf1b13ca35cce2e6bcf-00000000000000000000000000000000
I20260928 05:17:50.728313 6559 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 401619c49c134f59a36d3d960b3589ee: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 05:17:50.730376 7000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 05:17:50.736876 7000 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 109bdf09ab804bf1b13ca35cce2e6bcf: tablet copy: Starting download of 0 data blocks...
I20260928 05:17:50.737157 7000 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 109bdf09ab804bf1b13ca35cce2e6bcf: tablet copy: Starting download of 1 WAL segments...
I20260928 05:17:50.739539 7000 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 109bdf09ab804bf1b13ca35cce2e6bcf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 05:17:50.743558 6559 tablet_copy_service.cc:342] P 401619c49c134f59a36d3d960b3589ee: Request end of tablet copy session 109bdf09ab804bf1b13ca35cce2e6bcf-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:42556
I20260928 05:17:50.743714 6559 tablet_copy_service.cc:433] P 401619c49c134f59a36d3d960b3589ee: ending tablet copy session 109bdf09ab804bf1b13ca35cce2e6bcf-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 109bdf09ab804bf1b13ca35cce2e6bcf
I20260928 05:17:50.755023 7000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:17:50.757462 7109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:50.762050 7110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:17:50.766384 7000 server_base.cc:1034] running on GCE node
W20260928 05:17:50.766930 7112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:17:50.767377 7000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:17:50.767958 7000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:17:50.769135 7000 hybrid_clock.cc:648] HybridClock initialized: now 1790572670769092 us; error 30 us; skew 500 ppm
I20260928 05:17:50.769346 7000 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:17:50 dist-test-slave-whsn krb5kdc[6494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572670, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:17:50.771416 7000 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260928 05:17:50.772557 7000 webserver.cc:460] Webserver started at http://127.0.97.61:43359/ using document root <none> and password file <none>
I20260928 05:17:50.772925 7000 fs_manager.cc:359] Metadata directory not provided
I20260928 05:17:50.773015 7000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:17:50.776032 7000 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 05:17:50.777866 7119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:50.778262 7000 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 05:17:50.778380 7000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/master-1/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/master-1/wal
uuid: "109bdf09ab804bf1b13ca35cce2e6bcf"
format_stamp: "Formatted at 2026-09-28 05:17:50 on dist-test-slave-whsn"
server_key: "00ed56967467c1396881c08100e218dd"
server_key_iv: "bc982ab0c79e989cae5d765bdb9b90d5"
server_key_version: "encryptionkey@0"
I20260928 05:17:50.778532 7000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:17:50.792958 7000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:17:50.796705 7000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:17:50.818902 7000 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:40533
I20260928 05:17:50.818955 7170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:40533 every 8 connection(s)
I20260928 05:17:50.819832 7000 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/master-1/data/info.pb
I20260928 05:17:50.824729 7171 sys_catalog.cc:263] Verifying existing consensus state
I20260928 05:17:50.828291 7171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 109bdf09ab804bf1b13ca35cce2e6bcf: Bootstrap starting.
I20260928 05:17:50.847205 7171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 109bdf09ab804bf1b13ca35cce2e6bcf: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 05:17:50.847666 7171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 109bdf09ab804bf1b13ca35cce2e6bcf: Bootstrap complete.
I20260928 05:17:50.848675 7171 raft_consensus.cc:348] T 00000000000000000000000000000000 P 109bdf09ab804bf1b13ca35cce2e6bcf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "401619c49c134f59a36d3d960b3589ee" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38331 } } peers { permanent_uuid: "109bdf09ab804bf1b13ca35cce2e6bcf" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 40533 } attrs { promote: true } }
I20260928 05:17:50.848948 7171 raft_consensus.cc:729] T 00000000000000000000000000000000 P 109bdf09ab804bf1b13ca35cce2e6bcf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 109bdf09ab804bf1b13ca35cce2e6bcf, State: Initialized, Role: LEARNER
I20260928 05:17:50.849202 7171 consensus_queue.cc:260] T 00000000000000000000000000000000 P 109bdf09ab804bf1b13ca35cce2e6bcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "401619c49c134f59a36d3d960b3589ee" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38331 } } peers { permanent_uuid: "109bdf09ab804bf1b13ca35cce2e6bcf" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 40533 } attrs { promote: true } }
I20260928 05:17:50.851058 7174 sys_catalog.cc:455] T 00000000000000000000000000000000 P 109bdf09ab804bf1b13ca35cce2e6bcf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "401619c49c134f59a36d3d960b3589ee" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38331 } } peers { permanent_uuid: "109bdf09ab804bf1b13ca35cce2e6bcf" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 40533 } attrs { promote: true } } }
I20260928 05:17:50.851347 7171 sys_catalog.cc:564] T 00000000000000000000000000000000 P 109bdf09ab804bf1b13ca35cce2e6bcf [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:17:50.851503 7174 sys_catalog.cc:458] T 00000000000000000000000000000000 P 109bdf09ab804bf1b13ca35cce2e6bcf [sys.catalog]: This master's current role is: LEARNER
I20260928 05:17:50.863373 7185 catalog_manager.cc:1259] Loaded cluster ID: 617aa3f5fc464ce4a48eb676675ac7d8
I20260928 05:17:50.863471 7185 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 109bdf09ab804bf1b13ca35cce2e6bcf: loading cluster ID for follower catalog manager: success
I20260928 05:17:50.865913 7185 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 109bdf09ab804bf1b13ca35cce2e6bcf: acquiring CA information for follower catalog manager: success
I20260928 05:17:50.867633 7185 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 109bdf09ab804bf1b13ca35cce2e6bcf: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260928 05:17:51.035754 7100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:17:51.036016 7100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:17:51.036139 7100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:17:51.043391 7100 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:17:51.043496 7100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:17:51.043566 7100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:17:51.043637 7100 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:17:51.043692 7100 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:17:51.053772 7100 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38331,127.0.97.61:40533,127.0.97.60:39701
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:39701
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:17:51.054865 7100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:17:51.056022 7100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:17:51.066474 7190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:51.066730 7191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:51.067334 7193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:17:51.067567 7100 server_base.cc:1034] running on GCE node
I20260928 05:17:51.068221 7100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:17:51.068315 6995 heartbeater.cc:503] Master 127.0.97.62:38331 was elected leader, sending a full tablet report...
I20260928 05:17:51.069718 7100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:17:51.070917 7100 hybrid_clock.cc:648] HybridClock initialized: now 1790572671070863 us; error 53 us; skew 500 ppm
I20260928 05:17:51.071226 7100 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:17:51 dist-test-slave-whsn krb5kdc[6494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572671, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:17:51.073892 7100 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260928 05:17:51.075116 7100 webserver.cc:460] Webserver started at http://127.0.97.60:40499/ using document root <none> and password file <none>
I20260928 05:17:51.075721 7100 fs_manager.cc:359] Metadata directory not provided
I20260928 05:17:51.075824 7100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:17:51.076110 7100 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:17:51.078769 7100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/master-2/data/instance:
uuid: "599975ad9f374b9cba8b0d1e09b84822"
format_stamp: "Formatted at 2026-09-28 05:17:51 on dist-test-slave-whsn"
server_key: "759b0ed3658f0d87d783a5e903eeb9dd"
server_key_iv: "625d258a31f1c6ed3e1201fafc497105"
server_key_version: "encryptionkey@0"
I20260928 05:17:51.079439 7100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/master-2/wal/instance:
uuid: "599975ad9f374b9cba8b0d1e09b84822"
format_stamp: "Formatted at 2026-09-28 05:17:51 on dist-test-slave-whsn"
server_key: "759b0ed3658f0d87d783a5e903eeb9dd"
server_key_iv: "625d258a31f1c6ed3e1201fafc497105"
server_key_version: "encryptionkey@0"
I20260928 05:17:51.083850 7100 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260928 05:17:51.087303 7200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:51.088459 7100 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260928 05:17:51.088581 7100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/master-2/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/master-2/wal
uuid: "599975ad9f374b9cba8b0d1e09b84822"
format_stamp: "Formatted at 2026-09-28 05:17:51 on dist-test-slave-whsn"
server_key: "759b0ed3658f0d87d783a5e903eeb9dd"
server_key_iv: "625d258a31f1c6ed3e1201fafc497105"
server_key_version: "encryptionkey@0"
I20260928 05:17:51.088721 7100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:17:51.107898 7100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:17:51.111253 7100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:17:51.111449 7100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:17:51.129067 7100 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:39701
I20260928 05:17:51.129086 7251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:39701 every 8 connection(s)
I20260928 05:17:51.130292 7100 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/master-2/data/info.pb
I20260928 05:17:51.133626 7252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:17:51.136976 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 7100
I20260928 05:17:51.140391 7252 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38331 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 05:17:51 dist-test-slave-whsn krb5kdc[6494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572668, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 28 05:17:51 dist-test-slave-whsn krb5kdc[6494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572671, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260928 05:17:51.157691 6730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:51.143451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42566 (local address 127.0.97.62:38331)
0928 05:17:51.143627 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:51.143634 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:51.144181 (+ 547us) server_negotiation.cc:407] Connection header received
0928 05:17:51.144366 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:51.144370 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:51.144441 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:51.144527 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:51.145799 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:51.146489 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:51.148069 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:51.148452 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:51.148543 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:51.152477 (+ 3934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:51.152482 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:51.152485 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:51.152515 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:51.155083 (+ 2568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:51.155707 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:51.155711 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:51.155713 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:51.155843 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:51.156169 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:51.156173 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:51.156175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:51.156460 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:51.156571 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:51.157155 (+ 584us) server_negotiation.cc:299] Negotiation successful
0928 05:17:51.157360 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 05:17:51.158579 7255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:51.139049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45560 (local address 127.0.97.60:39701)
0928 05:17:51.139727 (+ 678us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:51.139738 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:51.139776 (+ 38us) server_negotiation.cc:407] Connection header received
0928 05:17:51.141352 (+ 1576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:51.141373 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:51.141748 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:51.142082 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:51.143513 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:51.145111 (+ 1598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:51.146533 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:51.147174 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:51.147286 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:51.152349 (+ 5063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:51.152359 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:51.152372 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:51.152403 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:51.154922 (+ 2519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:51.155574 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:51.155581 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:51.155585 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:51.155742 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:51.156138 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:51.156142 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:51.156146 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:51.156560 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:51.156803 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:51.157126 (+ 323us) server_negotiation.cc:299] Negotiation successful
0928 05:17:51.157397 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":493,"thread_start_us":309,"threads_started":1}
I20260928 05:17:51.158610 7256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:51.143332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38331 (local address 127.0.0.1:42566)
0928 05:17:51.144049 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:51.144086 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:51.144239 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:51.144606 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:51.144613 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:51.145318 (+ 705us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:17:51.145640 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:51.145654 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:51.146619 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:51.146623 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:51.147937 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:51.147949 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:51.148657 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:51.148665 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:51.148889 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:51.149931 (+ 1042us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:17:51.149963 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:17:51.152303 (+ 2340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:17:51.155197 (+ 2894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:51.155205 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:51.155212 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:51.155589 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:51.155938 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:51.155941 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:51.155944 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:51.156066 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:51.156591 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:17:51.156598 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:17:51.156774 (+ 176us) client_negotiation.cc:769] Sending connection context
0928 05:17:51.157052 (+ 278us) client_negotiation.cc:240] Negotiation successful
0928 05:17:51.157509 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"mutex_wait_us":95,"thread_start_us":377,"threads_started":1}
I20260928 05:17:51.160631 7252 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40533 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 05:17:51 dist-test-slave-whsn krb5kdc[6494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572671, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260928 05:17:51.172081 7256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:51.161243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40533 (local address 127.0.0.1:39612)
0928 05:17:51.161388 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:51.161403 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:51.161474 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:51.162017 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:51.162019 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:51.162350 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:17:51.162480 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:51.162489 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:51.163609 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:51.163614 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:51.164495 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:51.164506 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:51.165537 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:51.165542 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:51.165694 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:51.166131 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:17:51.166147 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:17:51.167845 (+ 1698us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:17:51.170328 (+ 2483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:51.170332 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:51.170334 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:51.170667 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:51.170936 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:51.170939 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:51.170941 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:51.171062 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:51.171463 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:17:51.171466 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:17:51.171578 (+ 112us) client_negotiation.cc:769] Sending connection context
0928 05:17:51.171687 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 05:17:51.171864 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260928 05:17:51.172118 7257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:51.161320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39612 (local address 127.0.97.61:40533)
0928 05:17:51.161792 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:51.161796 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:51.161811 (+ 15us) server_negotiation.cc:407] Connection header received
0928 05:17:51.161855 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:51.161858 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:51.161909 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:51.161990 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:51.162589 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:51.163481 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:51.164594 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:51.165403 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:51.165498 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:51.168006 (+ 2508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:51.168011 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:51.168014 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:51.168040 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:51.170217 (+ 2177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:51.170750 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:51.170753 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:51.170755 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:51.170860 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:51.171143 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:51.171146 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:51.171148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:51.171355 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:51.171458 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:51.171754 (+ 296us) server_negotiation.cc:299] Negotiation successful
0928 05:17:51.171891 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":264,"threads_started":1}
I20260928 05:17:51.173095 7252 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39701 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 05:17:51 dist-test-slave-whsn krb5kdc[6494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572671, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260928 05:17:51.183364 7256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:51.173608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39701 (local address 127.0.0.1:45574)
0928 05:17:51.173754 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:51.173771 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:51.173852 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:51.174085 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:51.174088 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:51.174372 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:17:51.174518 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:51.174525 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:51.175333 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:51.175337 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:51.176195 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:51.176207 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:51.176784 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:51.176789 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:51.177005 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:51.177404 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:17:51.177419 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:17:51.179094 (+ 1675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:17:51.181448 (+ 2354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:51.181452 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:51.181454 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:51.181815 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:51.182112 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:51.182115 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:51.182118 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:51.182255 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:51.182644 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:17:51.182647 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:17:51.182754 (+ 107us) client_negotiation.cc:769] Sending connection context
0928 05:17:51.182862 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 05:17:51.183025 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260928 05:17:51.183369 7255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:51.173626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45574 (local address 127.0.97.60:39701)
0928 05:17:51.173747 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:51.173751 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:51.173811 (+ 60us) server_negotiation.cc:407] Connection header received
0928 05:17:51.173933 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:51.173936 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:51.173989 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:51.174056 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:51.174624 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:51.175232 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:51.176307 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:51.176660 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:51.176743 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:51.179246 (+ 2503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:51.179251 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:51.179254 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:51.179278 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:51.181335 (+ 2057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:51.181907 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:51.181910 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:51.181912 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:51.182024 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:51.182345 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:51.182348 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:51.182350 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:51.182560 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:51.182650 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:51.182915 (+ 265us) server_negotiation.cc:299] Negotiation successful
0928 05:17:51.183116 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":62,"server-negotiator.queue_time_us":37}
I20260928 05:17:51.187824 7252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 599975ad9f374b9cba8b0d1e09b84822: Bootstrap starting.
I20260928 05:17:51.190557 7252 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 599975ad9f374b9cba8b0d1e09b84822: Neither blocks nor log segments found. Creating new log.
I20260928 05:17:51.191461 7252 log.cc:824] T 00000000000000000000000000000000 P 599975ad9f374b9cba8b0d1e09b84822: Log is configured to *not* fsync() on all Append() calls
I20260928 05:17:51.193744 7252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 599975ad9f374b9cba8b0d1e09b84822: No bootstrap required, opened a new log
I20260928 05:17:51.200011 7252 raft_consensus.cc:348] T 00000000000000000000000000000000 P 599975ad9f374b9cba8b0d1e09b84822 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401619c49c134f59a36d3d960b3589ee" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38331 } } peers { permanent_uuid: "109bdf09ab804bf1b13ca35cce2e6bcf" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40533 } } peers { permanent_uuid: "599975ad9f374b9cba8b0d1e09b84822" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39701 } }
I20260928 05:17:51.200266 7252 raft_consensus.cc:374] T 00000000000000000000000000000000 P 599975ad9f374b9cba8b0d1e09b84822 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:17:51.200364 7252 raft_consensus.cc:729] T 00000000000000000000000000000000 P 599975ad9f374b9cba8b0d1e09b84822 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 599975ad9f374b9cba8b0d1e09b84822, State: Initialized, Role: FOLLOWER
I20260928 05:17:51.200891 7252 consensus_queue.cc:260] T 00000000000000000000000000000000 P 599975ad9f374b9cba8b0d1e09b84822 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401619c49c134f59a36d3d960b3589ee" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38331 } } peers { permanent_uuid: "109bdf09ab804bf1b13ca35cce2e6bcf" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40533 } } peers { permanent_uuid: "599975ad9f374b9cba8b0d1e09b84822" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39701 } }
I20260928 05:17:51.202351 7259 sys_catalog.cc:455] T 00000000000000000000000000000000 P 599975ad9f374b9cba8b0d1e09b84822 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401619c49c134f59a36d3d960b3589ee" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38331 } } peers { permanent_uuid: "109bdf09ab804bf1b13ca35cce2e6bcf" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40533 } } peers { permanent_uuid: "599975ad9f374b9cba8b0d1e09b84822" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39701 } } }
I20260928 05:17:51.202652 7259 sys_catalog.cc:458] T 00000000000000000000000000000000 P 599975ad9f374b9cba8b0d1e09b84822 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 05:17:51.203203 7252 sys_catalog.cc:564] T 00000000000000000000000000000000 P 599975ad9f374b9cba8b0d1e09b84822 [sys.catalog]: configured and running, proceeding with master startup.
W20260928 05:17:51.210847 7270 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 599975ad9f374b9cba8b0d1e09b84822: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 05:17:51.210964 7270 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 05:17:51.212615 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 6600
I20260928 05:17:51.215692 7100 master_runner.cc:428] Detected that this master 599975ad9f374b9cba8b0d1e09b84822 is joining an existing cluster
I20260928 05:17:51.215791 7100 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.60:39701
Sep 28 05:17:51 dist-test-slave-whsn krb5kdc[6494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572668, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
I20260928 05:17:51.243476 7273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:51.214284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34320 (local address 127.0.97.1:43633)
0928 05:17:51.221359 (+ 7075us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:51.221369 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:51.221401 (+ 32us) server_negotiation.cc:407] Connection header received
0928 05:17:51.221477 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:51.221486 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:51.221742 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:51.223630 (+ 1888us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:51.223896 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:51.225777 (+ 1881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:51.228077 (+ 2300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:51.228922 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:51.234767 (+ 5845us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:51.235066 (+ 299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:51.235077 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:51.235095 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:51.235128 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:51.239801 (+ 4673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:51.240599 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:51.240607 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:51.240611 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:51.240744 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:51.241177 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:51.241181 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:51.241184 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:51.241573 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:51.242714 (+ 1141us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:51.242868 (+ 154us) server_negotiation.cc:299] Negotiation successful
0928 05:17:51.243100 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6886,"thread_start_us":6736,"threads_started":1}
I20260928 05:17:51.243541 7257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:51.214640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39620 (local address 127.0.97.61:40533)
0928 05:17:51.215349 (+ 709us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:51.215354 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:51.236961 (+ 21607us) server_negotiation.cc:407] Connection header received
0928 05:17:51.237247 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:51.237253 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:51.237359 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:51.237439 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:17:51.238049 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:51.239028 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:51.241271 (+ 2243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:51.242402 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:51.242500 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:51.242707 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:51.243217 (+ 510us) server_negotiation.cc:299] Negotiation successful
0928 05:17:51.243341 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593}
I20260928 05:17:51.244347 7274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:51.214448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40533 (local address 127.0.0.1:39620)
0928 05:17:51.236869 (+ 22421us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:51.236888 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:51.237090 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:51.237648 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:51.237653 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:51.237680 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:17:51.237876 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:51.237887 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:51.239205 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:51.239210 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:51.241118 (+ 1908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:51.241132 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:51.242845 (+ 1713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:51.242851 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:51.243017 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:51.243073 (+ 56us) client_negotiation.cc:769] Sending connection context
0928 05:17:51.243921 (+ 848us) client_negotiation.cc:240] Negotiation successful
0928 05:17:51.244057 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22303,"thread_start_us":22200,"threads_started":1}
I20260928 05:17:51.244983 6723 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:17:51.245535 7146 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 109bdf09ab804bf1b13ca35cce2e6bcf [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260928 05:17:51.246274 7174 sys_catalog.cc:455] T 00000000000000000000000000000000 P 109bdf09ab804bf1b13ca35cce2e6bcf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 401619c49c134f59a36d3d960b3589ee. Latest consensus state: current_term: 1 leader_uuid: "401619c49c134f59a36d3d960b3589ee" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "401619c49c134f59a36d3d960b3589ee" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38331 } } peers { permanent_uuid: "109bdf09ab804bf1b13ca35cce2e6bcf" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 40533 } attrs { promote: true } } }
I20260928 05:17:51.246524 7174 sys_catalog.cc:458] T 00000000000000000000000000000000 P 109bdf09ab804bf1b13ca35cce2e6bcf [sys.catalog]: This master's current role is: LEARNER
I20260928 05:17:51.276046 6730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:51.250705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42570 (local address 127.0.97.62:38331)
0928 05:17:51.250857 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:51.250862 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:51.254530 (+ 3668us) server_negotiation.cc:407] Connection header received
0928 05:17:51.254782 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:51.254787 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:51.254860 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:51.254959 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:51.256255 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:51.257355 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:51.265224 (+ 7869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:51.265879 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:51.265967 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:51.268472 (+ 2505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:51.268479 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:51.268484 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:51.268535 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:51.272824 (+ 4289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:51.273847 (+ 1023us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:51.273852 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:51.273856 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:51.273997 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:51.274429 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:51.274434 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:51.274438 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:51.274739 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:51.274845 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:51.275471 (+ 626us) server_negotiation.cc:299] Negotiation successful
0928 05:17:51.275690 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 05:17:51.277000 7281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:51.250318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38331 (local address 127.0.0.1:42570)
0928 05:17:51.254429 (+ 4111us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:51.254449 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:51.254638 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:51.255001 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:51.255007 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:51.255720 (+ 713us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:17:51.256093 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:51.256105 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:51.257525 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:51.257534 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:51.265065 (+ 7531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:51.265080 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:51.266216 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:51.266223 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:51.266410 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:51.267136 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:17:51.267159 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:17:51.268271 (+ 1112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:17:51.272997 (+ 4726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:51.273003 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:51.273006 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:51.273724 (+ 718us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:51.274128 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:51.274133 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:51.274137 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:51.274312 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:51.274930 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:17:51.274938 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:17:51.275147 (+ 209us) client_negotiation.cc:769] Sending connection context
0928 05:17:51.276228 (+ 1081us) client_negotiation.cc:240] Negotiation successful
0928 05:17:51.276532 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3960,"mutex_wait_us":2,"thread_start_us":14563,"threads_started":1}
I20260928 05:17:51.277751 7255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:51.262914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45580 (local address 127.0.97.60:39701)
0928 05:17:51.263087 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:51.263092 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:51.263109 (+ 17us) server_negotiation.cc:407] Connection header received
0928 05:17:51.263150 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:51.263155 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:51.263221 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:51.263302 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:51.265149 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:51.266208 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:51.268149 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:51.268823 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:51.268939 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:51.271076 (+ 2137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:51.271080 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:51.271083 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:51.271107 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:51.275033 (+ 3926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:51.275718 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:51.275723 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:51.275726 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:51.275859 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:51.276283 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:51.276288 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:51.276292 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:51.276629 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:51.276728 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:51.277249 (+ 521us) server_negotiation.cc:299] Negotiation successful
0928 05:17:51.277430 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 05:17:51.279263 7282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:51.253797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39701 (local address 127.0.0.1:45580)
0928 05:17:51.262784 (+ 8987us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:51.262804 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:51.262922 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:51.263727 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:51.263732 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:51.264367 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:17:51.264714 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:51.264726 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:51.266420 (+ 1694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:51.266428 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:51.268014 (+ 1586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:51.268028 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:51.269192 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:51.269199 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:51.269377 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:51.269898 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:17:51.269919 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:17:51.270950 (+ 1031us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:17:51.275162 (+ 4212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:51.275166 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:51.275170 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:51.275604 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:51.275967 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:51.275971 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:51.275974 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:51.276099 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:51.276802 (+ 703us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:17:51.276806 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:17:51.276999 (+ 193us) client_negotiation.cc:769] Sending connection context
0928 05:17:51.278331 (+ 1332us) client_negotiation.cc:240] Negotiation successful
0928 05:17:51.278602 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8832,"mutex_wait_us":24,"thread_start_us":320,"threads_started":1}
I20260928 05:17:51.291687 7280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:51.257428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40533 (local address 127.0.0.1:39632)
0928 05:17:51.265140 (+ 7712us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:51.265156 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:51.265312 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:51.271640 (+ 6328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:51.271643 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:51.272572 (+ 929us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:17:51.272904 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:51.272917 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:51.274243 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:51.274251 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:51.275815 (+ 1564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:51.275830 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:51.277605 (+ 1775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:51.277614 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:51.277774 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:51.284970 (+ 7196us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:17:51.284986 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:17:51.285586 (+ 600us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:17:51.289054 (+ 3468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:51.289060 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:51.289068 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:51.289482 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:51.289875 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:51.289880 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:51.289884 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:51.290060 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:51.290637 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:17:51.290643 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:17:51.290838 (+ 195us) client_negotiation.cc:769] Sending connection context
0928 05:17:51.290972 (+ 134us) client_negotiation.cc:240] Negotiation successful
0928 05:17:51.291230 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7596,"thread_start_us":5020,"threads_started":1}
I20260928 05:17:51.291723 7257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:51.253572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39632 (local address 127.0.97.61:40533)
0928 05:17:51.253706 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:51.253710 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:51.265224 (+ 11514us) server_negotiation.cc:407] Connection header received
0928 05:17:51.271331 (+ 6107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:51.271337 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:51.271417 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:51.271484 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:51.273082 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:51.274074 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:51.275958 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:51.276600 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:51.276692 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:51.285763 (+ 9071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:51.285769 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:51.285775 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:51.285813 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:51.288934 (+ 3121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:51.289617 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:51.289623 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:51.289626 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:51.289767 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:51.290173 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:51.290178 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:51.290181 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:51.290518 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:51.290656 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:51.291103 (+ 447us) server_negotiation.cc:299] Negotiation successful
0928 05:17:51.291322 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
Sep 28 05:17:51 dist-test-slave-whsn krb5kdc[6494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572668, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260928 05:17:51.314844 7274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:51.283300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39701 (local address 127.0.0.1:45596)
0928 05:17:51.283450 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:51.283466 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:51.283563 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:51.283999 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:51.284004 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:51.284421 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:17:51.284598 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:51.284608 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:51.302410 (+ 17802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:51.302416 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:51.303819 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:51.303835 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:51.304898 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:51.304905 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:51.305100 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:51.305724 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:17:51.305746 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:17:51.308359 (+ 2613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:17:51.312089 (+ 3730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:51.312095 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:51.312099 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:51.312638 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:51.313045 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:51.313051 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:51.313055 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:51.313252 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:51.313889 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:17:51.313895 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:17:51.314111 (+ 216us) client_negotiation.cc:769] Sending connection context
0928 05:17:51.314243 (+ 132us) client_negotiation.cc:240] Negotiation successful
0928 05:17:51.314507 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260928 05:17:51.315775 7255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:51.283543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45596 (local address 127.0.97.60:39701)
0928 05:17:51.283681 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:51.283687 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:51.283702 (+ 15us) server_negotiation.cc:407] Connection header received
0928 05:17:51.283777 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:51.283782 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:51.283879 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:51.283976 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:51.301075 (+ 17099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:51.302228 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:51.303954 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:51.304723 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:51.304835 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:51.308557 (+ 3722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:51.308563 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:51.308568 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:51.308604 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:51.311959 (+ 3355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:51.312777 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:51.312783 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:51.312786 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:51.312927 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:51.313365 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:51.313369 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:51.313373 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:51.313691 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:51.315064 (+ 1373us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:51.315173 (+ 109us) server_negotiation.cc:299] Negotiation successful
0928 05:17:51.315419 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 05:17:51.317396 6539 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.60:39701: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "599975ad9f374b9cba8b0d1e09b84822"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.60"
port: 39701
}
attrs {
promote: true
}
}
dest_uuid: "401619c49c134f59a36d3d960b3589ee"
cas_config_opid_index: 5
I20260928 05:17:51.318444 6539 consensus_queue.cc:237] T 00000000000000000000000000000000 P 401619c49c134f59a36d3d960b3589ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "401619c49c134f59a36d3d960b3589ee" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38331 } } peers { permanent_uuid: "109bdf09ab804bf1b13ca35cce2e6bcf" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 40533 } attrs { promote: true } } peers { permanent_uuid: "599975ad9f374b9cba8b0d1e09b84822" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 39701 } attrs { promote: true } }
I20260928 05:17:51.320374 7272 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 401619c49c134f59a36d3d960b3589ee [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 599975ad9f374b9cba8b0d1e09b84822 (127.0.97.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "401619c49c134f59a36d3d960b3589ee" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38331 } } peers { permanent_uuid: "109bdf09ab804bf1b13ca35cce2e6bcf" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 40533 } attrs { promote: true } } peers { permanent_uuid: "599975ad9f374b9cba8b0d1e09b84822" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 39701 } attrs { promote: true } } }
I20260928 05:17:51.324100 7272 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.60:39701
I20260928 05:17:51.325117 7146 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 109bdf09ab804bf1b13ca35cce2e6bcf [term 1 LEARNER]: Refusing update from remote peer 401619c49c134f59a36d3d960b3589ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 05:17:51.326203 7100 master.cc:537] Master@127.0.97.60:39701 shutting down...
I20260928 05:17:51.339010 7100 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 599975ad9f374b9cba8b0d1e09b84822 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260928 05:17:51.339390 7100 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 599975ad9f374b9cba8b0d1e09b84822 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260928 05:17:51.339578 7100 tablet_replica.cc:330] T 00000000000000000000000000000000 P 599975ad9f374b9cba8b0d1e09b84822: stopping tablet replica
W20260928 05:17:51.339566 6526 consensus_peers.cc:487] T 00000000000000000000000000000000 P 401619c49c134f59a36d3d960b3589ee -> Peer 599975ad9f374b9cba8b0d1e09b84822 (127.0.97.60:39701): Couldn't send request to peer 599975ad9f374b9cba8b0d1e09b84822. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 05:17:51.353260 7284 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 401619c49c134f59a36d3d960b3589ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "109bdf09ab804bf1b13ca35cce2e6bcf" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 40533 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260928 05:17:51.356580 7100 master.cc:552] Master@127.0.97.60:39701 shutdown complete.
I20260928 05:17:51.356667 7100 master_runner.cc:305] Clearing existing system tablet
I20260928 05:17:51.357084 7100 fs_manager.cc:359] Metadata directory not provided
I20260928 05:17:51.357183 7100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:17:51.359627 7146 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 109bdf09ab804bf1b13ca35cce2e6bcf [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 599975ad9f374b9cba8b0d1e09b84822 (127.0.97.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "401619c49c134f59a36d3d960b3589ee" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38331 } } peers { permanent_uuid: "109bdf09ab804bf1b13ca35cce2e6bcf" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 40533 } attrs { promote: true } } peers { permanent_uuid: "599975ad9f374b9cba8b0d1e09b84822" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 39701 } attrs { promote: true } } }
I20260928 05:17:51.361322 7100 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260928 05:17:51.363916 7174 sys_catalog.cc:455] T 00000000000000000000000000000000 P 109bdf09ab804bf1b13ca35cce2e6bcf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "401619c49c134f59a36d3d960b3589ee" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "401619c49c134f59a36d3d960b3589ee" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38331 } } peers { permanent_uuid: "109bdf09ab804bf1b13ca35cce2e6bcf" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 40533 } attrs { promote: true } } peers { permanent_uuid: "599975ad9f374b9cba8b0d1e09b84822" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 39701 } attrs { promote: true } } }
I20260928 05:17:51.364146 7174 sys_catalog.cc:458] T 00000000000000000000000000000000 P 109bdf09ab804bf1b13ca35cce2e6bcf [sys.catalog]: This master's current role is: LEARNER
I20260928 05:17:51.365027 7286 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 401619c49c134f59a36d3d960b3589ee: attempting to promote NON_VOTER 109bdf09ab804bf1b13ca35cce2e6bcf to VOTER
I20260928 05:17:51.365139 7283 sys_catalog.cc:455] T 00000000000000000000000000000000 P 401619c49c134f59a36d3d960b3589ee [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "401619c49c134f59a36d3d960b3589ee" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "401619c49c134f59a36d3d960b3589ee" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38331 } } peers { permanent_uuid: "109bdf09ab804bf1b13ca35cce2e6bcf" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 40533 } attrs { promote: true } } peers { permanent_uuid: "599975ad9f374b9cba8b0d1e09b84822" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 39701 } attrs { promote: true } } }
I20260928 05:17:51.365492 7283 sys_catalog.cc:458] T 00000000000000000000000000000000 P 401619c49c134f59a36d3d960b3589ee [sys.catalog]: This master's current role is: LEADER
I20260928 05:17:51.366032 7289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:51.365836 7286 consensus_queue.cc:237] T 00000000000000000000000000000000 P 401619c49c134f59a36d3d960b3589ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "401619c49c134f59a36d3d960b3589ee" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38331 } } peers { permanent_uuid: "109bdf09ab804bf1b13ca35cce2e6bcf" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40533 } attrs { promote: false } } peers { permanent_uuid: "599975ad9f374b9cba8b0d1e09b84822" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 39701 } attrs { promote: true } }
I20260928 05:17:51.366369 7100 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:17:51.366590 7100 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 05:17:51.366683 7100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/master-2/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/master-2/wal
uuid: "599975ad9f374b9cba8b0d1e09b84822"
format_stamp: "Formatted at 2026-09-28 05:17:51 on dist-test-slave-whsn"
server_key: "759b0ed3658f0d87d783a5e903eeb9dd"
server_key_iv: "625d258a31f1c6ed3e1201fafc497105"
server_key_version: "encryptionkey@0"
I20260928 05:17:51.367838 7274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:51.367340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39701 (local address 127.0.0.1:45610)
0928 05:17:51.367513 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:51.367659 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260928 05:17:51.368287 7146 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 109bdf09ab804bf1b13ca35cce2e6bcf [term 1 LEARNER]: Refusing update from remote peer 401619c49c134f59a36d3d960b3589ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260928 05:17:51.368782 7284 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 401619c49c134f59a36d3d960b3589ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "109bdf09ab804bf1b13ca35cce2e6bcf" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40533 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260928 05:17:51.369096 7274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:51.368614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39701 (local address 127.0.0.1:45624)
0928 05:17:51.368811 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:51.368905 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
W20260928 05:17:51.369349 6526 consensus_peers.cc:487] T 00000000000000000000000000000000 P 401619c49c134f59a36d3d960b3589ee -> Peer 599975ad9f374b9cba8b0d1e09b84822 (127.0.97.60:39701): Couldn't send request to peer 599975ad9f374b9cba8b0d1e09b84822. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39701: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 05:17:51.370810 7286 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 401619c49c134f59a36d3d960b3589ee [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 109bdf09ab804bf1b13ca35cce2e6bcf (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "401619c49c134f59a36d3d960b3589ee" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38331 } } peers { permanent_uuid: "109bdf09ab804bf1b13ca35cce2e6bcf" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40533 } attrs { promote: false } } peers { permanent_uuid: "599975ad9f374b9cba8b0d1e09b84822" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 39701 } attrs { promote: true } } }
I20260928 05:17:51.371429 7100 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 599975ad9f374b9cba8b0d1e09b84822: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:17:51.371487 7146 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 109bdf09ab804bf1b13ca35cce2e6bcf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 109bdf09ab804bf1b13ca35cce2e6bcf (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "401619c49c134f59a36d3d960b3589ee" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38331 } } peers { permanent_uuid: "109bdf09ab804bf1b13ca35cce2e6bcf" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40533 } attrs { promote: false } } peers { permanent_uuid: "599975ad9f374b9cba8b0d1e09b84822" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 39701 } attrs { promote: true } } }
I20260928 05:17:51.375029 7174 sys_catalog.cc:455] T 00000000000000000000000000000000 P 109bdf09ab804bf1b13ca35cce2e6bcf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "401619c49c134f59a36d3d960b3589ee" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "401619c49c134f59a36d3d960b3589ee" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38331 } } peers { permanent_uuid: "109bdf09ab804bf1b13ca35cce2e6bcf" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40533 } attrs { promote: false } } peers { permanent_uuid: "599975ad9f374b9cba8b0d1e09b84822" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 39701 } attrs { promote: true } } }
I20260928 05:17:51.375259 7174 sys_catalog.cc:458] T 00000000000000000000000000000000 P 109bdf09ab804bf1b13ca35cce2e6bcf [sys.catalog]: This master's current role is: FOLLOWER
I20260928 05:17:51.375523 7286 sys_catalog.cc:455] T 00000000000000000000000000000000 P 401619c49c134f59a36d3d960b3589ee [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "401619c49c134f59a36d3d960b3589ee" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "401619c49c134f59a36d3d960b3589ee" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38331 } } peers { permanent_uuid: "109bdf09ab804bf1b13ca35cce2e6bcf" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40533 } attrs { promote: false } } peers { permanent_uuid: "599975ad9f374b9cba8b0d1e09b84822" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 39701 } attrs { promote: true } } }
I20260928 05:17:51.376127 7286 sys_catalog.cc:458] T 00000000000000000000000000000000 P 401619c49c134f59a36d3d960b3589ee [sys.catalog]: This master's current role is: LEADER
I20260928 05:17:51.375869 7272 sys_catalog.cc:455] T 00000000000000000000000000000000 P 401619c49c134f59a36d3d960b3589ee [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "401619c49c134f59a36d3d960b3589ee" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "401619c49c134f59a36d3d960b3589ee" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38331 } } peers { permanent_uuid: "109bdf09ab804bf1b13ca35cce2e6bcf" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40533 } attrs { promote: false } } peers { permanent_uuid: "599975ad9f374b9cba8b0d1e09b84822" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 39701 } attrs { promote: true } } }
I20260928 05:17:51.376365 7272 sys_catalog.cc:458] T 00000000000000000000000000000000 P 401619c49c134f59a36d3d960b3589ee [sys.catalog]: This master's current role is: LEADER
I20260928 05:17:51.381388 7100 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 599975ad9f374b9cba8b0d1e09b84822: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 05:17:51.381534 7100 log.cc:1196] T 00000000000000000000000000000000 P 599975ad9f374b9cba8b0d1e09b84822: Deleting WAL directory at /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260928 05:17:51.382005 7100 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 599975ad9f374b9cba8b0d1e09b84822: Deleting consensus metadata
I20260928 05:17:51.382231 7100 master_runner.cc:315] Copying system tablet from 127.0.97.62:38331
I20260928 05:17:51.384778 7100 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 599975ad9f374b9cba8b0d1e09b84822: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:38331
I20260928 05:17:51.386562 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 6600
I20260928 05:17:51.399235 6730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:51.385538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42580 (local address 127.0.97.62:38331)
0928 05:17:51.386156 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:51.386162 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:51.386181 (+ 19us) server_negotiation.cc:407] Connection header received
0928 05:17:51.386233 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:51.386238 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:51.386306 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:51.386417 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:51.387748 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:51.388912 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:51.390692 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:51.391303 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:51.391427 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:51.393494 (+ 2067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:51.393500 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:51.393505 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:51.393539 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:51.396581 (+ 3042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:51.397315 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:51.397320 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:51.397324 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:51.397471 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:51.397856 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:51.397860 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:51.397864 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:51.398172 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:51.398302 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:51.398702 (+ 400us) server_negotiation.cc:299] Negotiation successful
0928 05:17:51.398917 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":494}
I20260928 05:17:51.399312 7296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:51.385499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38331 (local address 127.0.0.1:42580)
0928 05:17:51.385958 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:51.385980 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:51.386106 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:51.386453 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:51.386458 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:51.387342 (+ 884us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:17:51.387577 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:51.387590 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:51.389091 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:51.389096 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:51.390545 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:51.390563 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:51.391449 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:51.391455 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:51.391616 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:51.392257 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:17:51.392279 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:17:51.393292 (+ 1013us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:17:51.396704 (+ 3412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:51.396710 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:51.396713 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:51.397212 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:51.397581 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:51.397586 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:51.397590 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:51.397747 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:51.398282 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:17:51.398287 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:17:51.398448 (+ 161us) client_negotiation.cc:769] Sending connection context
0928 05:17:51.398597 (+ 149us) client_negotiation.cc:240] Negotiation successful
0928 05:17:51.398890 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":219,"threads_started":1}
I20260928 05:17:51.400151 6559 tablet_copy_service.cc:140] P 401619c49c134f59a36d3d960b3589ee: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 599975ad9f374b9cba8b0d1e09b84822 ({username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:42580)
I20260928 05:17:51.400305 6559 tablet_copy_service.cc:161] P 401619c49c134f59a36d3d960b3589ee: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 599975ad9f374b9cba8b0d1e09b84822 at {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:42580: session id = 599975ad9f374b9cba8b0d1e09b84822-00000000000000000000000000000000
I20260928 05:17:51.403393 6559 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 401619c49c134f59a36d3d960b3589ee: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 05:17:51.403630 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:43633
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:38331
--webserver_port=34703
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:38331,127.0.97.61:40533
--tserver_master_addrs=127.0.97.62:38331,127.0.97.61:40533,127.0.97.60:39701 with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 05:17:51.405093 7100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 05:17:51.409464 7100 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 599975ad9f374b9cba8b0d1e09b84822: tablet copy: Starting download of 0 data blocks...
I20260928 05:17:51.409691 7100 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 599975ad9f374b9cba8b0d1e09b84822: tablet copy: Starting download of 1 WAL segments...
I20260928 05:17:51.411844 7100 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 599975ad9f374b9cba8b0d1e09b84822: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 05:17:51.416003 6559 tablet_copy_service.cc:342] P 401619c49c134f59a36d3d960b3589ee: Request end of tablet copy session 599975ad9f374b9cba8b0d1e09b84822-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:42580
I20260928 05:17:51.416126 6559 tablet_copy_service.cc:433] P 401619c49c134f59a36d3d960b3589ee: ending tablet copy session 599975ad9f374b9cba8b0d1e09b84822-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 599975ad9f374b9cba8b0d1e09b84822
I20260928 05:17:51.427574 7100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:17:51.429957 7298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:51.436255 7299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:51.440214 7301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:17:51.443212 7100 server_base.cc:1034] running on GCE node
I20260928 05:17:51.443542 7100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:17:51.444125 7100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:17:51.445389 7100 hybrid_clock.cc:648] HybridClock initialized: now 1790572671445344 us; error 28 us; skew 500 ppm
I20260928 05:17:51.445616 7100 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:17:51 dist-test-slave-whsn krb5kdc[6494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572671, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:17:51.447661 7100 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260928 05:17:51.452175 7100 webserver.cc:460] Webserver started at http://127.0.97.60:35181/ using document root <none> and password file <none>
I20260928 05:17:51.452539 7100 fs_manager.cc:359] Metadata directory not provided
I20260928 05:17:51.452636 7100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:17:51.455631 7100 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 05:17:51.457458 7308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:51.457891 7100 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260928 05:17:51.458066 7100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/master-2/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/master-2/wal
uuid: "599975ad9f374b9cba8b0d1e09b84822"
format_stamp: "Formatted at 2026-09-28 05:17:51 on dist-test-slave-whsn"
server_key: "759b0ed3658f0d87d783a5e903eeb9dd"
server_key_iv: "625d258a31f1c6ed3e1201fafc497105"
server_key_version: "encryptionkey@0"
I20260928 05:17:51.458280 7100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:17:51.478499 7100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:17:51.482507 7100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:17:51.505975 7100 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:39701
I20260928 05:17:51.506048 7359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:39701 every 8 connection(s)
I20260928 05:17:51.507109 7100 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/master-2/data/info.pb
I20260928 05:17:51.512300 7360 sys_catalog.cc:263] Verifying existing consensus state
I20260928 05:17:51.515973 7360 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 599975ad9f374b9cba8b0d1e09b84822: Bootstrap starting.
I20260928 05:17:51.533555 7360 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 599975ad9f374b9cba8b0d1e09b84822: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 05:17:51.533999 7360 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 599975ad9f374b9cba8b0d1e09b84822: Bootstrap complete.
I20260928 05:17:51.534969 7360 raft_consensus.cc:348] T 00000000000000000000000000000000 P 599975ad9f374b9cba8b0d1e09b84822 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "401619c49c134f59a36d3d960b3589ee" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38331 } } peers { permanent_uuid: "109bdf09ab804bf1b13ca35cce2e6bcf" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40533 } attrs { promote: false } } peers { permanent_uuid: "599975ad9f374b9cba8b0d1e09b84822" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 39701 } attrs { promote: true } }
I20260928 05:17:51.535247 7360 raft_consensus.cc:729] T 00000000000000000000000000000000 P 599975ad9f374b9cba8b0d1e09b84822 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 599975ad9f374b9cba8b0d1e09b84822, State: Initialized, Role: LEARNER
I20260928 05:17:51.535497 7360 consensus_queue.cc:260] T 00000000000000000000000000000000 P 599975ad9f374b9cba8b0d1e09b84822 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "401619c49c134f59a36d3d960b3589ee" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38331 } } peers { permanent_uuid: "109bdf09ab804bf1b13ca35cce2e6bcf" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40533 } attrs { promote: false } } peers { permanent_uuid: "599975ad9f374b9cba8b0d1e09b84822" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 39701 } attrs { promote: true } }
I20260928 05:17:51.536500 7360 sys_catalog.cc:564] T 00000000000000000000000000000000 P 599975ad9f374b9cba8b0d1e09b84822 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:17:51.536242 7363 sys_catalog.cc:455] T 00000000000000000000000000000000 P 599975ad9f374b9cba8b0d1e09b84822 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "401619c49c134f59a36d3d960b3589ee" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38331 } } peers { permanent_uuid: "109bdf09ab804bf1b13ca35cce2e6bcf" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40533 } attrs { promote: false } } peers { permanent_uuid: "599975ad9f374b9cba8b0d1e09b84822" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 39701 } attrs { promote: true } } }
I20260928 05:17:51.536746 7363 sys_catalog.cc:458] T 00000000000000000000000000000000 P 599975ad9f374b9cba8b0d1e09b84822 [sys.catalog]: This master's current role is: LEARNER
I20260928 05:17:51.548702 7374 catalog_manager.cc:1259] Loaded cluster ID: 617aa3f5fc464ce4a48eb676675ac7d8
I20260928 05:17:51.548805 7374 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 599975ad9f374b9cba8b0d1e09b84822: loading cluster ID for follower catalog manager: success
I20260928 05:17:51.552225 7374 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 599975ad9f374b9cba8b0d1e09b84822: acquiring CA information for follower catalog manager: success
I20260928 05:17:51.554498 7374 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 599975ad9f374b9cba8b0d1e09b84822: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260928 05:17:51.747301 7297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:17:51.747543 7297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:17:51.747656 7297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:17:51.754602 7297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:17:51.754693 7297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:17:51.754830 7297 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 05:17:51.764369 7297 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:43633
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=34703
--tserver_master_addrs=127.0.97.62:38331,127.0.97.61:40533,127.0.97.60:39701
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:17:51.765573 7297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:17:51.766785 7297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:17:51.778977 7382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:51.779004 7380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:51.779110 7379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:17:51.780158 7297 server_base.cc:1034] running on GCE node
I20260928 05:17:51.780678 7297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:17:51.781527 7297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:17:51.782724 7297 hybrid_clock.cc:648] HybridClock initialized: now 1790572671782678 us; error 35 us; skew 500 ppm
I20260928 05:17:51.783037 7297 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:17:51 dist-test-slave-whsn krb5kdc[6494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572671, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:17:51.785873 7297 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260928 05:17:51.787191 7297 webserver.cc:460] Webserver started at http://127.0.97.1:34703/ using document root <none> and password file <none>
I20260928 05:17:51.787781 7297 fs_manager.cc:359] Metadata directory not provided
I20260928 05:17:51.787894 7297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:17:51.793375 7297 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 05:17:51.796938 7390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:51.798204 7297 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 05:17:51.798595 7385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:51.782890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45636 (local address 127.0.97.60:39701)
0928 05:17:51.793296 (+ 10406us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:51.793303 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:51.793325 (+ 22us) server_negotiation.cc:407] Connection header received
0928 05:17:51.793399 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:51.793404 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:51.793501 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:51.793635 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:17:51.793994 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:51.795341 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:51.796865 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:51.797650 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:51.797766 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:51.797968 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:51.798154 (+ 186us) server_negotiation.cc:299] Negotiation successful
0928 05:17:51.798275 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10279,"thread_start_us":10135,"threads_started":1}
I20260928 05:17:51.798595 7274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:51.782762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39701 (local address 127.0.0.1:45636)
0928 05:17:51.782933 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:51.782953 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:51.783074 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:51.793671 (+ 10597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:51.793675 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:51.793703 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:17:51.793859 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:51.793867 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:51.795508 (+ 1641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:51.795513 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:51.796707 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:51.796723 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:51.797818 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:51.797823 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:51.798016 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:51.798041 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 05:17:51.798123 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 05:17:51.798275 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 05:17:51.798343 7297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/ts-0/wal
uuid: "dafeb19659594f5abdc1fc79acdfb9df"
format_stamp: "Formatted at 2026-09-28 05:17:49 on dist-test-slave-whsn"
server_key: "7a12a4b12aad7e4e1f4ff9c36a343b9d"
server_key_iv: "44848d1908588bee0444a5c3b11c713b"
server_key_version: "encryptionkey@0"
I20260928 05:17:51.799428 7297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:17:51.799677 7335 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 599975ad9f374b9cba8b0d1e09b84822 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260928 05:17:51.800258 7363 sys_catalog.cc:455] T 00000000000000000000000000000000 P 599975ad9f374b9cba8b0d1e09b84822 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 401619c49c134f59a36d3d960b3589ee. Latest consensus state: current_term: 1 leader_uuid: "401619c49c134f59a36d3d960b3589ee" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "401619c49c134f59a36d3d960b3589ee" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38331 } } peers { permanent_uuid: "109bdf09ab804bf1b13ca35cce2e6bcf" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40533 } attrs { promote: false } } peers { permanent_uuid: "599975ad9f374b9cba8b0d1e09b84822" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 39701 } attrs { promote: true } } }
I20260928 05:17:51.800506 7363 sys_catalog.cc:458] T 00000000000000000000000000000000 P 599975ad9f374b9cba8b0d1e09b84822 [sys.catalog]: This master's current role is: LEARNER
I20260928 05:17:51.821280 7297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:17:51.826352 7297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:17:51.826624 7297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:17:51.827634 7297 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:17:51.829241 7297 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:17:51.829339 7297 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:51.829428 7297 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:17:51.829494 7297 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:51.861049 7297 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43633
I20260928 05:17:51.861104 7502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43633 every 8 connection(s)
I20260928 05:17:51.862358 7297 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/ts-0/data/info.pb
I20260928 05:17:51.867676 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 7297
I20260928 05:17:51.867981 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 6734
Sep 28 05:17:51 dist-test-slave-whsn krb5kdc[6494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572671, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Sep 28 05:17:51 dist-test-slave-whsn krb5kdc[6494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572668, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
I20260928 05:17:51.889439 6730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:51.865313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49447 (local address 127.0.97.62:38331)
0928 05:17:51.865651 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:51.865658 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:51.866847 (+ 1189us) server_negotiation.cc:407] Connection header received
0928 05:17:51.870013 (+ 3166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:51.870020 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:51.870104 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:51.870176 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:51.872392 (+ 2216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:51.873498 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:51.876011 (+ 2513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:51.876753 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:51.876869 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:51.882063 (+ 5194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:51.882068 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:51.882073 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:51.882110 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:51.885418 (+ 3308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:51.886289 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:51.886294 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:51.886297 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:51.886449 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:51.887414 (+ 965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:51.887419 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:51.887422 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:51.887750 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:51.887859 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:51.888888 (+ 1029us) server_negotiation.cc:299] Negotiation successful
0928 05:17:51.889099 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192}
Sep 28 05:17:51 dist-test-slave-whsn krb5kdc[6494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572671, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 28 05:17:51 dist-test-slave-whsn krb5kdc[6494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572671, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260928 05:17:51.892488 7507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:51.865872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38331 (local address 127.0.97.1:49447)
0928 05:17:51.866697 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:51.866746 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:51.869716 (+ 2970us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:51.870708 (+ 992us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:51.870719 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:51.871434 (+ 715us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:17:51.872199 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:51.872221 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:51.873656 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:51.873663 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:51.875835 (+ 2172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:51.875851 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:51.877463 (+ 1612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:51.877470 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:51.877795 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:51.878651 (+ 856us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:17:51.878683 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:17:51.881830 (+ 3147us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:17:51.885602 (+ 3772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:51.885610 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:51.885625 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:51.886167 (+ 542us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:51.886610 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:51.886615 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:51.886619 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:51.887282 (+ 663us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:51.887965 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:17:51.887972 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:17:51.888369 (+ 397us) client_negotiation.cc:769] Sending connection context
0928 05:17:51.890550 (+ 2181us) client_negotiation.cc:240] Negotiation successful
0928 05:17:51.891046 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":3379,"threads_started":1}
I20260928 05:17:51.894296 7504 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38331
I20260928 05:17:51.894619 7504 heartbeater.cc:464] Registering TS with master...
I20260928 05:17:51.895593 7504 heartbeater.cc:511] Master 127.0.97.62:38331 requested a full tablet report, sending...
I20260928 05:17:51.896291 7385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:51.865417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52531 (local address 127.0.97.60:39701)
0928 05:17:51.865608 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:51.865614 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:51.880406 (+ 14792us) server_negotiation.cc:407] Connection header received
0928 05:17:51.880633 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:51.880639 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:51.880715 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:51.880778 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:51.882041 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:51.883047 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:51.885367 (+ 2320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:51.885980 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:51.886078 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:51.890421 (+ 4343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:51.890427 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:51.890432 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:51.890469 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:51.893473 (+ 3004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:51.894267 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:51.894272 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:51.894275 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:51.894428 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:51.894879 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:51.894884 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:51.894888 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:51.895194 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:51.895310 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:51.895801 (+ 491us) server_negotiation.cc:299] Negotiation successful
0928 05:17:51.896004 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 05:17:51.897239 7509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:51.865908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39701 (local address 127.0.97.1:52531)
0928 05:17:51.880316 (+ 14408us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:51.880332 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:51.880497 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:51.880990 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:51.880994 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:51.881575 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:17:51.881859 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:51.881869 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:51.883227 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:51.883232 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:51.885210 (+ 1978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:51.885226 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:51.887079 (+ 1853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:51.887088 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:51.887296 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:51.887808 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:17:51.887829 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:17:51.890232 (+ 2403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:17:51.893628 (+ 3396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:51.893633 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:51.893637 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:51.894150 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:51.894580 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:51.894584 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:51.894588 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:51.894758 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:51.895385 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:17:51.895388 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:17:51.895549 (+ 161us) client_negotiation.cc:769] Sending connection context
0928 05:17:51.896518 (+ 969us) client_negotiation.cc:240] Negotiation successful
0928 05:17:51.896809 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14291,"mutex_wait_us":1,"spinlock_wait_cycles":19328,"thread_start_us":17268,"threads_started":1}
I20260928 05:17:51.897253 6539 ts_manager.cc:194] Re-registered known tserver with Master: dafeb19659594f5abdc1fc79acdfb9df (127.0.97.1:43633)
I20260928 05:17:51.897985 7512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:51.869238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41032 (local address 127.0.97.2:33089)
0928 05:17:51.873109 (+ 3871us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:51.873117 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:51.875677 (+ 2560us) server_negotiation.cc:407] Connection header received
0928 05:17:51.875901 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:51.876216 (+ 315us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:51.876463 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:51.876606 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:51.879012 (+ 2406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:51.881203 (+ 2191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:51.883353 (+ 2150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:51.884128 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:51.884248 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:51.889743 (+ 5495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:51.889755 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:51.889771 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:51.890056 (+ 285us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:51.894070 (+ 4014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:51.895138 (+ 1068us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:51.895146 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:51.895149 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:51.895287 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:51.895695 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:51.895700 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:51.895703 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:51.896089 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:51.896352 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:51.897406 (+ 1054us) server_negotiation.cc:299] Negotiation successful
0928 05:17:51.897635 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3719,"thread_start_us":3552,"threads_started":1}
I20260928 05:17:51.898272 6539 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:49447
I20260928 05:17:51.898959 7503 heartbeater.cc:347] Connected to a master server at 127.0.97.60:39701
I20260928 05:17:51.899096 7503 heartbeater.cc:464] Registering TS with master...
I20260928 05:17:51.899475 7503 heartbeater.cc:511] Master 127.0.97.60:39701 requested a full tablet report, sending...
I20260928 05:17:51.899550 7510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:51.867079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56291 (local address 127.0.97.61:40533)
0928 05:17:51.872679 (+ 5600us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:51.872683 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:51.883667 (+ 10984us) server_negotiation.cc:407] Connection header received
0928 05:17:51.883904 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:51.883909 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:51.883990 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:51.884083 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:51.885331 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:51.886318 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:51.888537 (+ 2219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:51.889112 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:51.889199 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:51.892642 (+ 3443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:51.892648 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:51.892654 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:51.892688 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:51.896231 (+ 3543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:51.897221 (+ 990us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:51.897225 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:51.897228 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:51.897354 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:51.897953 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:51.897958 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:51.897962 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:51.898256 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:51.898371 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:51.898997 (+ 626us) server_negotiation.cc:299] Negotiation successful
0928 05:17:51.899183 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5473,"thread_start_us":5361,"threads_started":1}
I20260928 05:17:51.899585 7508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:51.866559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40533 (local address 127.0.97.1:56291)
0928 05:17:51.883570 (+ 17011us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:51.883591 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:51.883764 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:51.884307 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:51.884312 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:51.884833 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:17:51.885174 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:51.885185 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:51.886475 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:51.886480 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:51.888387 (+ 1907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:51.888399 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:51.889430 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:51.889435 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:51.889586 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:51.890085 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:17:51.890104 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:17:51.892411 (+ 2307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:17:51.896579 (+ 4168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:51.896583 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:51.896586 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:51.897097 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:51.897686 (+ 589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:51.897691 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:51.897694 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:51.897837 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:51.898634 (+ 797us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:17:51.898638 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:17:51.898772 (+ 134us) client_negotiation.cc:769] Sending connection context
0928 05:17:51.898874 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 05:17:51.899184 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16848,"thread_start_us":13388,"threads_started":1}
I20260928 05:17:51.900095 6857 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:17:51.900442 7505 heartbeater.cc:347] Connected to a master server at 127.0.97.61:40533
I20260928 05:17:51.900570 7505 heartbeater.cc:464] Registering TS with master...
I20260928 05:17:51.900923 7505 heartbeater.cc:511] Master 127.0.97.61:40533 requested a full tablet report, sending...
I20260928 05:17:51.901892 7325 ts_manager.cc:194] Registered new tserver with Master: dafeb19659594f5abdc1fc79acdfb9df (127.0.97.1:43633)
I20260928 05:17:51.903501 7136 ts_manager.cc:194] Registered new tserver with Master: dafeb19659594f5abdc1fc79acdfb9df (127.0.97.1:43633)
I20260928 05:17:51.983145 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 6734
I20260928 05:17:51.995819 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:33089
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:38331
--webserver_port=45031
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:38331,127.0.97.61:40533
--tserver_master_addrs=127.0.97.62:38331,127.0.97.61:40533,127.0.97.60:39701 with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 05:17:52.259346 7272 sys_catalog.cc:455] T 00000000000000000000000000000000 P 401619c49c134f59a36d3d960b3589ee [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "401619c49c134f59a36d3d960b3589ee" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "401619c49c134f59a36d3d960b3589ee" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38331 } } peers { permanent_uuid: "109bdf09ab804bf1b13ca35cce2e6bcf" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40533 } attrs { promote: false } } peers { permanent_uuid: "599975ad9f374b9cba8b0d1e09b84822" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 39701 } attrs { promote: true } } }
I20260928 05:17:52.259697 7272 sys_catalog.cc:458] T 00000000000000000000000000000000 P 401619c49c134f59a36d3d960b3589ee [sys.catalog]: This master's current role is: LEADER
I20260928 05:17:52.260905 7272 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 401619c49c134f59a36d3d960b3589ee: attempting to promote NON_VOTER 599975ad9f374b9cba8b0d1e09b84822 to VOTER
I20260928 05:17:52.261579 7272 consensus_queue.cc:237] T 00000000000000000000000000000000 P 401619c49c134f59a36d3d960b3589ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "401619c49c134f59a36d3d960b3589ee" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38331 } } peers { permanent_uuid: "109bdf09ab804bf1b13ca35cce2e6bcf" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40533 } attrs { promote: false } } peers { permanent_uuid: "599975ad9f374b9cba8b0d1e09b84822" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39701 } attrs { promote: false } }
I20260928 05:17:52.263502 7146 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 109bdf09ab804bf1b13ca35cce2e6bcf [term 1 FOLLOWER]: Refusing update from remote peer 401619c49c134f59a36d3d960b3589ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260928 05:17:52.263847 7335 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 599975ad9f374b9cba8b0d1e09b84822 [term 1 LEARNER]: Refusing update from remote peer 401619c49c134f59a36d3d960b3589ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260928 05:17:52.264122 7516 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 401619c49c134f59a36d3d960b3589ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "109bdf09ab804bf1b13ca35cce2e6bcf" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40533 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260928 05:17:52.264515 7272 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 401619c49c134f59a36d3d960b3589ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "599975ad9f374b9cba8b0d1e09b84822" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39701 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260928 05:17:52.267568 7516 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 401619c49c134f59a36d3d960b3589ee [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 599975ad9f374b9cba8b0d1e09b84822 (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "401619c49c134f59a36d3d960b3589ee" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38331 } } peers { permanent_uuid: "109bdf09ab804bf1b13ca35cce2e6bcf" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40533 } attrs { promote: false } } peers { permanent_uuid: "599975ad9f374b9cba8b0d1e09b84822" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39701 } attrs { promote: false } } }
I20260928 05:17:52.268312 7146 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 109bdf09ab804bf1b13ca35cce2e6bcf [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 599975ad9f374b9cba8b0d1e09b84822 (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "401619c49c134f59a36d3d960b3589ee" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38331 } } peers { permanent_uuid: "109bdf09ab804bf1b13ca35cce2e6bcf" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40533 } attrs { promote: false } } peers { permanent_uuid: "599975ad9f374b9cba8b0d1e09b84822" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39701 } attrs { promote: false } } }
I20260928 05:17:52.271348 7515 sys_catalog.cc:455] T 00000000000000000000000000000000 P 401619c49c134f59a36d3d960b3589ee [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "401619c49c134f59a36d3d960b3589ee" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "401619c49c134f59a36d3d960b3589ee" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38331 } } peers { permanent_uuid: "109bdf09ab804bf1b13ca35cce2e6bcf" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40533 } attrs { promote: false } } peers { permanent_uuid: "599975ad9f374b9cba8b0d1e09b84822" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39701 } attrs { promote: false } } }
I20260928 05:17:52.271636 7515 sys_catalog.cc:458] T 00000000000000000000000000000000 P 401619c49c134f59a36d3d960b3589ee [sys.catalog]: This master's current role is: LEADER
I20260928 05:17:52.271565 7516 sys_catalog.cc:455] T 00000000000000000000000000000000 P 401619c49c134f59a36d3d960b3589ee [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "401619c49c134f59a36d3d960b3589ee" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "401619c49c134f59a36d3d960b3589ee" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38331 } } peers { permanent_uuid: "109bdf09ab804bf1b13ca35cce2e6bcf" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40533 } attrs { promote: false } } peers { permanent_uuid: "599975ad9f374b9cba8b0d1e09b84822" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39701 } attrs { promote: false } } }
I20260928 05:17:52.271898 7516 sys_catalog.cc:458] T 00000000000000000000000000000000 P 401619c49c134f59a36d3d960b3589ee [sys.catalog]: This master's current role is: LEADER
I20260928 05:17:52.271924 7515 sys_catalog.cc:455] T 00000000000000000000000000000000 P 401619c49c134f59a36d3d960b3589ee [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "401619c49c134f59a36d3d960b3589ee" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "401619c49c134f59a36d3d960b3589ee" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38331 } } peers { permanent_uuid: "109bdf09ab804bf1b13ca35cce2e6bcf" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40533 } attrs { promote: false } } peers { permanent_uuid: "599975ad9f374b9cba8b0d1e09b84822" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39701 } attrs { promote: false } } }
I20260928 05:17:52.272133 7515 sys_catalog.cc:458] T 00000000000000000000000000000000 P 401619c49c134f59a36d3d960b3589ee [sys.catalog]: This master's current role is: LEADER
I20260928 05:17:52.272912 7519 sys_catalog.cc:455] T 00000000000000000000000000000000 P 109bdf09ab804bf1b13ca35cce2e6bcf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "401619c49c134f59a36d3d960b3589ee" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "401619c49c134f59a36d3d960b3589ee" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38331 } } peers { permanent_uuid: "109bdf09ab804bf1b13ca35cce2e6bcf" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40533 } attrs { promote: false } } peers { permanent_uuid: "599975ad9f374b9cba8b0d1e09b84822" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39701 } attrs { promote: false } } }
I20260928 05:17:52.273309 7519 sys_catalog.cc:458] T 00000000000000000000000000000000 P 109bdf09ab804bf1b13ca35cce2e6bcf [sys.catalog]: This master's current role is: FOLLOWER
I20260928 05:17:52.272787 7335 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 599975ad9f374b9cba8b0d1e09b84822 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 599975ad9f374b9cba8b0d1e09b84822 (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "401619c49c134f59a36d3d960b3589ee" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38331 } } peers { permanent_uuid: "109bdf09ab804bf1b13ca35cce2e6bcf" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40533 } attrs { promote: false } } peers { permanent_uuid: "599975ad9f374b9cba8b0d1e09b84822" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39701 } attrs { promote: false } } }
I20260928 05:17:52.275841 7363 sys_catalog.cc:455] T 00000000000000000000000000000000 P 599975ad9f374b9cba8b0d1e09b84822 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "401619c49c134f59a36d3d960b3589ee" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "401619c49c134f59a36d3d960b3589ee" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38331 } } peers { permanent_uuid: "109bdf09ab804bf1b13ca35cce2e6bcf" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40533 } attrs { promote: false } } peers { permanent_uuid: "599975ad9f374b9cba8b0d1e09b84822" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39701 } attrs { promote: false } } }
I20260928 05:17:52.276081 7363 sys_catalog.cc:458] T 00000000000000000000000000000000 P 599975ad9f374b9cba8b0d1e09b84822 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 05:17:52.298533 7514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:17:52.298988 7514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:17:52.299124 7514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:17:52.306259 7514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:17:52.306357 7514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:17:52.306512 7514 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 05:17:52.315505 7514 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:33089
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=45031
--tserver_master_addrs=127.0.97.62:38331,127.0.97.61:40533,127.0.97.60:39701
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:17:52.316589 7514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:17:52.317713 7514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:17:52.329294 7525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:52.329254 7527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:52.329334 7524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:17:52.330005 7514 server_base.cc:1034] running on GCE node
I20260928 05:17:52.330504 7514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:17:52.331280 7514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:17:52.332487 7514 hybrid_clock.cc:648] HybridClock initialized: now 1790572672332434 us; error 43 us; skew 500 ppm
I20260928 05:17:52.332769 7514 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:17:52 dist-test-slave-whsn krb5kdc[6494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572672, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:17:52.335561 7514 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260928 05:17:52.336796 7514 webserver.cc:460] Webserver started at http://127.0.97.2:45031/ using document root <none> and password file <none>
I20260928 05:17:52.337379 7514 fs_manager.cc:359] Metadata directory not provided
I20260928 05:17:52.337507 7514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:17:52.342192 7514 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 05:17:52.344595 7534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:52.345870 7514 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 05:17:52.345997 7514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/ts-1/wal
uuid: "176eae27f2854c7a84d8a0e5c342d0a9"
format_stamp: "Formatted at 2026-09-28 05:17:49 on dist-test-slave-whsn"
server_key: "39e5977ad88887f426576145a904f05c"
server_key_iv: "d8e71ec1cd5a98733d0e1dda27b80903"
server_key_version: "encryptionkey@0"
I20260928 05:17:52.346861 7514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:17:52.366329 7514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:17:52.369958 7514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:17:52.370186 7514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:17:52.371037 7514 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:17:52.372215 7514 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:17:52.372298 7514 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:52.372402 7514 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:17:52.372474 7514 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:52.400713 7514 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33089
I20260928 05:17:52.400770 7646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33089 every 8 connection(s)
I20260928 05:17:52.402086 7514 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/ts-1/data/info.pb
I20260928 05:17:52.403757 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 7514
I20260928 05:17:52.404000 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 6867
Sep 28 05:17:52 dist-test-slave-whsn krb5kdc[6494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572672, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 28 05:17:52 dist-test-slave-whsn krb5kdc[6494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572672, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Sep 28 05:17:52 dist-test-slave-whsn krb5kdc[6494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572672, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Sep 28 05:17:52 dist-test-slave-whsn krb5kdc[6494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572668, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
I20260928 05:17:52.452809 7652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:52.405484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43305 (local address 127.0.97.60:39701)
0928 05:17:52.429718 (+ 24234us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:52.429725 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:52.429748 (+ 23us) server_negotiation.cc:407] Connection header received
0928 05:17:52.431613 (+ 1865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:52.431620 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:52.431703 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:52.431796 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:52.434026 (+ 2230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:52.435138 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:52.438506 (+ 3368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:52.439090 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:52.439204 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:52.443957 (+ 4753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:52.443964 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:52.443968 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:52.444006 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:52.448936 (+ 4930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:52.449836 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:52.449842 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:52.449846 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:52.450001 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:52.450471 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:52.450476 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:52.450479 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:52.450804 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:52.451060 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:52.452176 (+ 1116us) server_negotiation.cc:299] Negotiation successful
0928 05:17:52.452370 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24110,"thread_start_us":23984,"threads_started":1}
I20260928 05:17:52.454581 7654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:52.405739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57901 (local address 127.0.97.61:40533)
0928 05:17:52.430072 (+ 24333us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:52.430078 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:52.438603 (+ 8525us) server_negotiation.cc:407] Connection header received
0928 05:17:52.438802 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:52.438807 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:52.438899 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:52.438980 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:52.440399 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:52.441511 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:52.443509 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:52.444081 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:52.444176 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:52.448109 (+ 3933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:52.448115 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:52.448120 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:52.448163 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:52.451562 (+ 3399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:52.452352 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:52.452356 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:52.452360 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:52.452478 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:52.452909 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:52.452914 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:52.452917 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:52.453292 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:52.453406 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:52.453979 (+ 573us) server_negotiation.cc:299] Negotiation successful
0928 05:17:52.454233 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24212,"thread_start_us":24342,"threads_started":1}
I20260928 05:17:52.454705 7656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:52.405494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39701 (local address 127.0.97.2:43305)
0928 05:17:52.426977 (+ 21483us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:52.427026 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:52.431182 (+ 4156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:52.432207 (+ 1025us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:52.432221 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:52.433026 (+ 805us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:17:52.433835 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:52.433859 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:52.435350 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:52.435356 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:52.437448 (+ 2092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:52.437462 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:52.439525 (+ 2063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:52.439532 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:52.439814 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:52.440647 (+ 833us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:17:52.440676 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:17:52.443731 (+ 3055us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:17:52.449107 (+ 5376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:52.449116 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:52.449135 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:52.449709 (+ 574us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:52.450122 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:52.450128 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:52.450131 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:52.450340 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:52.451130 (+ 790us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:17:52.451138 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:17:52.451509 (+ 371us) client_negotiation.cc:769] Sending connection context
0928 05:17:52.451845 (+ 336us) client_negotiation.cc:240] Negotiation successful
0928 05:17:52.453324 (+ 1479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21150,"mutex_wait_us":25,"thread_start_us":32086,"threads_started":1}
I20260928 05:17:52.455775 7655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:52.405503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40533 (local address 127.0.97.2:57901)
0928 05:17:52.438515 (+ 33012us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:52.438535 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:52.438681 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:52.439216 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:52.439221 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:52.439841 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:17:52.440201 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:52.440213 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:52.441662 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:52.441667 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:52.443366 (+ 1699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:52.443379 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:52.444461 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:52.444467 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:52.444661 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:52.445160 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:17:52.445181 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:17:52.447904 (+ 2723us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:17:52.451724 (+ 3820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:52.451730 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:52.451733 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:52.452232 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:52.452627 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:52.452632 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:52.452635 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:52.452766 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:52.453507 (+ 741us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:17:52.453511 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:17:52.453656 (+ 145us) client_negotiation.cc:769] Sending connection context
0928 05:17:52.454808 (+ 1152us) client_negotiation.cc:240] Negotiation successful
0928 05:17:52.455052 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32890,"mutex_wait_us":382,"thread_start_us":22993,"threads_started":1}
I20260928 05:17:52.456197 7647 heartbeater.cc:347] Connected to a master server at 127.0.97.60:39701
I20260928 05:17:52.456503 7647 heartbeater.cc:464] Registering TS with master...
I20260928 05:17:52.456569 7649 heartbeater.cc:347] Connected to a master server at 127.0.97.61:40533
I20260928 05:17:52.456691 7649 heartbeater.cc:464] Registering TS with master...
I20260928 05:17:52.457230 7651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:52.405224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60773 (local address 127.0.97.62:38331)
0928 05:17:52.439887 (+ 34663us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:52.439892 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:52.439911 (+ 19us) server_negotiation.cc:407] Connection header received
0928 05:17:52.439978 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:52.439983 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:52.440055 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:52.440714 (+ 659us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:52.442155 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:52.443781 (+ 1626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:52.445553 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:52.446130 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:52.446237 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:52.449561 (+ 3324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:52.449567 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:52.449572 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:52.449607 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:52.453270 (+ 3663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:52.454001 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:52.454006 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:52.454009 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:52.454135 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:52.454537 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:52.454542 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:52.454545 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:52.455641 (+ 1096us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:52.455771 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:52.456665 (+ 894us) server_negotiation.cc:299] Negotiation successful
0928 05:17:52.456874 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34552,"thread_start_us":35850,"threads_started":1}
I20260928 05:17:52.457396 7649 heartbeater.cc:511] Master 127.0.97.61:40533 requested a full tablet report, sending...
I20260928 05:17:52.457770 7647 heartbeater.cc:511] Master 127.0.97.60:39701 requested a full tablet report, sending...
I20260928 05:17:52.457793 7653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:52.405494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38331 (local address 127.0.97.2:60773)
0928 05:17:52.437961 (+ 32467us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:52.437978 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:52.438075 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:52.440891 (+ 2816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:52.440896 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:52.441548 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:17:52.441972 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:52.441984 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:52.443916 (+ 1932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:52.443920 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:52.445362 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:52.445379 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:52.446318 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:52.446325 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:52.446500 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:52.447085 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:17:52.447106 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:17:52.449352 (+ 2246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:17:52.453401 (+ 4049us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:52.453406 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:52.453409 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:52.453866 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:52.454251 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:52.454256 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:52.454259 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:52.454406 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:52.455823 (+ 1417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:17:52.455827 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:17:52.455984 (+ 157us) client_negotiation.cc:769] Sending connection context
0928 05:17:52.456106 (+ 122us) client_negotiation.cc:240] Negotiation successful
0928 05:17:52.456332 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32313,"mutex_wait_us":69,"spinlock_wait_cycles":15232,"thread_start_us":32606,"threads_started":1}
I20260928 05:17:52.458631 7648 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38331
I20260928 05:17:52.458756 7648 heartbeater.cc:464] Registering TS with master...
I20260928 05:17:52.459115 7136 ts_manager.cc:194] Registered new tserver with Master: 176eae27f2854c7a84d8a0e5c342d0a9 (127.0.97.2:33089)
I20260928 05:17:52.459128 7648 heartbeater.cc:511] Master 127.0.97.62:38331 requested a full tablet report, sending...
I20260928 05:17:52.459462 7325 ts_manager.cc:194] Registered new tserver with Master: 176eae27f2854c7a84d8a0e5c342d0a9 (127.0.97.2:33089)
I20260928 05:17:52.460086 6539 ts_manager.cc:194] Re-registered known tserver with Master: 176eae27f2854c7a84d8a0e5c342d0a9 (127.0.97.2:33089)
I20260928 05:17:52.460209 7657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:52.406329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51982 (local address 127.0.97.3:46223)
0928 05:17:52.438918 (+ 32589us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:52.438926 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:52.438964 (+ 38us) server_negotiation.cc:407] Connection header received
0928 05:17:52.439042 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:52.439050 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:52.439279 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:52.439441 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:52.441646 (+ 2205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:52.443271 (+ 1625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:52.445503 (+ 2232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:52.446557 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:52.446668 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:52.452248 (+ 5580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:52.452260 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:52.452276 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:52.452312 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:52.456355 (+ 4043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:52.457131 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:52.457139 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:52.457142 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:52.457277 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:52.457734 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:52.457738 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:52.457741 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:52.458163 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:52.459571 (+ 1408us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:52.459718 (+ 147us) server_negotiation.cc:299] Negotiation successful
0928 05:17:52.459943 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32432,"thread_start_us":32280,"threads_started":1}
I20260928 05:17:52.460788 6539 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:60773
I20260928 05:17:52.461344 6990 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:17:52.546017 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 6867
I20260928 05:17:52.558876 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:46223
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:38331
--webserver_port=39949
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:38331,127.0.97.61:40533
--tserver_master_addrs=127.0.97.62:38331,127.0.97.61:40533,127.0.97.60:39701 with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:17:52.857846 7659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:17:52.858094 7659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:17:52.858227 7659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:17:52.865459 7659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:17:52.865554 7659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:17:52.865692 7659 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 05:17:52.874730 7659 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:46223
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=39949
--tserver_master_addrs=127.0.97.62:38331,127.0.97.61:40533,127.0.97.60:39701
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:17:52.875808 7659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:17:52.876966 7659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:17:52.888674 7664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:52.888633 7665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:52.888710 7667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:17:52.889523 7659 server_base.cc:1034] running on GCE node
I20260928 05:17:52.890012 7659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:17:52.890810 7659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:17:52.891990 7659 hybrid_clock.cc:648] HybridClock initialized: now 1790572672891941 us; error 43 us; skew 500 ppm
I20260928 05:17:52.892251 7659 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:17:52 dist-test-slave-whsn krb5kdc[6494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572672, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:17:52.895031 7659 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260928 05:17:52.896195 7659 webserver.cc:460] Webserver started at http://127.0.97.3:39949/ using document root <none> and password file <none>
I20260928 05:17:52.896766 7659 fs_manager.cc:359] Metadata directory not provided
I20260928 05:17:52.896895 7659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:17:52.901597 7659 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260928 05:17:52.902077 7504 heartbeater.cc:503] Master 127.0.97.62:38331 was elected leader, sending a full tablet report...
I20260928 05:17:52.904593 7674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:52.905887 7659 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 05:17:52.906013 7659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/ts-2/wal
uuid: "a2d63277ac00458a83c7cb684be08738"
format_stamp: "Formatted at 2026-09-28 05:17:49 on dist-test-slave-whsn"
server_key: "e0fd083962b0f78af493447106d4353c"
server_key_iv: "7b8d610014d92e5e40b85449ecd4a27c"
server_key_version: "encryptionkey@0"
I20260928 05:17:52.906852 7659 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:17:52.946594 7659 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:17:52.950350 7659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:17:52.950577 7659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:17:52.951449 7659 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:17:52.952664 7659 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:17:52.952744 7659 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:52.952839 7659 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:17:52.952932 7659 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260928 05:17:52.981159 7659 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46223
I20260928 05:17:52.981230 7786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46223 every 8 connection(s)
I20260928 05:17:52.982445 7659 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/ts-2/data/info.pb
I20260928 05:17:52.984599 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 7659
Sep 28 05:17:53 dist-test-slave-whsn krb5kdc[6494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572672, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Sep 28 05:17:53 dist-test-slave-whsn krb5kdc[6494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572672, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260928 05:17:53.023761 7794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:52.990663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40533 (local address 127.0.97.3:45153)
0928 05:17:53.002985 (+ 12322us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:53.003005 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:53.003198 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:53.003813 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:53.003876 (+ 63us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:53.004475 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:17:53.005047 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:53.005073 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:53.006514 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:53.006519 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:53.008637 (+ 2118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:53.008655 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:53.010104 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:53.010110 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:53.010629 (+ 519us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:53.012465 (+ 1836us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:17:53.012486 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:17:53.015100 (+ 2614us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:17:53.019062 (+ 3962us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:53.019072 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:53.019091 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:53.019673 (+ 582us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:53.020125 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:53.020130 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:53.020134 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:53.020356 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:53.021186 (+ 830us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:17:53.021195 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:17:53.021615 (+ 420us) client_negotiation.cc:769] Sending connection context
0928 05:17:53.021930 (+ 315us) client_negotiation.cc:240] Negotiation successful
0928 05:17:53.022453 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12146,"mutex_wait_us":363,"thread_start_us":548,"threads_started":1}
I20260928 05:17:53.024411 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:52.991097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45153 (local address 127.0.97.61:40533)
0928 05:17:52.997266 (+ 6169us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:52.997273 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:53.003093 (+ 5820us) server_negotiation.cc:407] Connection header received
0928 05:17:53.003347 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:53.003353 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:53.003513 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:53.003601 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:53.005253 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:53.006335 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:53.008806 (+ 2471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:53.009420 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:53.009553 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:53.015256 (+ 5703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:53.015263 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:53.015268 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:53.015318 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:53.018885 (+ 3567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:53.019821 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:53.019827 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:53.019831 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:53.019986 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:53.020478 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:53.020482 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:53.020485 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:53.020797 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:53.020981 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:53.023920 (+ 2939us) server_negotiation.cc:299] Negotiation successful
0928 05:17:53.024105 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6036,"thread_start_us":5932,"threads_started":1}
I20260928 05:17:53.025734 7789 heartbeater.cc:347] Connected to a master server at 127.0.97.61:40533
I20260928 05:17:53.026316 7789 heartbeater.cc:464] Registering TS with master...
I20260928 05:17:53.027236 7789 heartbeater.cc:511] Master 127.0.97.61:40533 requested a full tablet report, sending...
I20260928 05:17:53.029052 7136 ts_manager.cc:194] Registered new tserver with Master: a2d63277ac00458a83c7cb684be08738 (127.0.97.3:46223)
I20260928 05:17:53.035137 7796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:52.985751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38331 (local address 127.0.97.3:58953)
0928 05:17:52.991488 (+ 5737us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:52.991547 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:52.994187 (+ 2640us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:53.003107 (+ 8920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:53.003120 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:53.003988 (+ 868us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:17:53.005047 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:53.005073 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:53.007412 (+ 2339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:53.007417 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:53.008882 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:53.008894 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:53.009826 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:53.009834 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:53.010097 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:53.010978 (+ 881us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:17:53.011007 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:17:53.014331 (+ 3324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:17:53.031807 (+ 17476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:53.031813 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:53.031817 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:53.032290 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:53.032719 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:53.032724 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:53.032728 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:53.033161 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:53.034194 (+ 1033us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:17:53.034200 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:17:53.034367 (+ 167us) client_negotiation.cc:769] Sending connection context
0928 05:17:53.034488 (+ 121us) client_negotiation.cc:240] Negotiation successful
0928 05:17:53.034744 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5393,"mutex_wait_us":7,"thread_start_us":16728,"threads_started":1}
Sep 28 05:17:53 dist-test-slave-whsn krb5kdc[6494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572672, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260928 05:17:53.036774 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:52.985562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58953 (local address 127.0.97.62:38331)
0928 05:17:53.001959 (+ 16397us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:53.001967 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:53.001987 (+ 20us) server_negotiation.cc:407] Connection header received
0928 05:17:53.002053 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:53.002058 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:53.002139 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:53.002861 (+ 722us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:53.005414 (+ 2553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:53.007292 (+ 1878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:53.008992 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:53.009625 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:53.009741 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:53.016881 (+ 7140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:53.016888 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:53.016892 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:53.016930 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:53.031662 (+ 14732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:53.032442 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:53.032448 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:53.032452 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:53.032599 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:53.033498 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:53.033506 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:53.033513 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:53.034054 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:53.035650 (+ 1596us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:53.035760 (+ 110us) server_negotiation.cc:299] Negotiation successful
0928 05:17:53.036022 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16272,"thread_start_us":16170,"threads_started":1}
I20260928 05:17:53.038424 7788 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38331
I20260928 05:17:53.038537 7788 heartbeater.cc:464] Registering TS with master...
I20260928 05:17:53.038924 7788 heartbeater.cc:511] Master 127.0.97.62:38331 requested a full tablet report, sending...
I20260928 05:17:53.039942 6539 ts_manager.cc:194] Re-registered known tserver with Master: a2d63277ac00458a83c7cb684be08738 (127.0.97.3:46223)
I20260928 05:17:53.041136 6539 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:58953
I20260928 05:17:53.044886 7791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:52.985239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39769 (local address 127.0.97.60:39701)
0928 05:17:53.002582 (+ 17343us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:53.002589 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:53.002606 (+ 17us) server_negotiation.cc:407] Connection header received
0928 05:17:53.002658 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:53.002663 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:53.002731 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:53.003528 (+ 797us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:53.005479 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:53.006523 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:53.030752 (+ 24229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:53.031452 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:53.031568 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:53.038535 (+ 6967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:53.038542 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:53.038547 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:53.038584 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:53.041688 (+ 3104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:53.042657 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:53.042662 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:53.042665 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:53.042821 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:53.043319 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:53.043324 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:53.043327 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:53.043625 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:53.043741 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:53.044254 (+ 513us) server_negotiation.cc:299] Negotiation successful
0928 05:17:53.044482 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":17210,"thread_start_us":17112,"threads_started":1}
I20260928 05:17:53.045895 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:52.986835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39701 (local address 127.0.97.3:39769)
0928 05:17:53.001397 (+ 14562us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:53.001416 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:53.001548 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:53.003107 (+ 1559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:53.003120 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:53.004589 (+ 1469us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:17:53.005330 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:53.005340 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:53.006674 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:53.006679 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:53.028464 (+ 21785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:53.028477 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:53.032890 (+ 4413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:53.032898 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:53.033076 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:53.035277 (+ 2201us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:17:53.035299 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:17:53.038291 (+ 2992us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:17:53.042054 (+ 3763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:53.042059 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:53.042062 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:53.042537 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:53.043022 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:53.043028 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:53.043031 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:53.043208 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:53.043812 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:17:53.043816 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:17:53.043979 (+ 163us) client_negotiation.cc:769] Sending connection context
0928 05:17:53.045234 (+ 1255us) client_negotiation.cc:240] Negotiation successful
0928 05:17:53.045514 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14433,"mutex_wait_us":20284,"thread_start_us":14238,"threads_started":1}
I20260928 05:17:53.045928 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:53.006598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60578 (local address 127.0.97.62:38331)
0928 05:17:53.020230 (+ 13632us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:53.020236 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:53.020253 (+ 17us) server_negotiation.cc:407] Connection header received
0928 05:17:53.020297 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:53.020302 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:53.020368 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:53.020458 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:53.022550 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:53.023636 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:53.029209 (+ 5573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:53.029794 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:53.029913 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:53.033172 (+ 3259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:53.033178 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:53.033182 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:53.033215 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:53.041015 (+ 7800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:53.041850 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:53.041854 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:53.041857 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:53.041992 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:53.044028 (+ 2036us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:53.044033 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:53.044037 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:53.044332 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:53.045293 (+ 961us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:53.045399 (+ 106us) server_negotiation.cc:299] Negotiation successful
0928 05:17:53.045588 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":664,"thread_start_us":574,"threads_started":1}
I20260928 05:17:53.046609 7787 heartbeater.cc:347] Connected to a master server at 127.0.97.60:39701
I20260928 05:17:53.046718 7787 heartbeater.cc:464] Registering TS with master...
I20260928 05:17:53.047057 7787 heartbeater.cc:511] Master 127.0.97.60:39701 requested a full tablet report, sending...
I20260928 05:17:53.047133 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:53.007971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51866 (local address 127.0.97.60:39701)
0928 05:17:53.023173 (+ 15202us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:53.023179 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:53.023810 (+ 631us) server_negotiation.cc:407] Connection header received
0928 05:17:53.023867 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:53.023872 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:53.023963 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:53.024058 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:53.027658 (+ 3600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:53.029177 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:53.031036 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:53.032039 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:53.032146 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:53.035638 (+ 3492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:53.035644 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:53.035648 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:53.035681 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:53.041966 (+ 6285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:53.043745 (+ 1779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:53.043749 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:53.043752 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:53.043873 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:53.044303 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:53.044308 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:53.044311 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:53.044620 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:53.044913 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:53.046639 (+ 1726us) server_negotiation.cc:299] Negotiation successful
0928 05:17:53.046825 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":15066,"thread_start_us":15577,"threads_started":1}
I20260928 05:17:53.048471 7324 ts_manager.cc:194] Registered new tserver with Master: a2d63277ac00458a83c7cb684be08738 (127.0.97.3:46223)
I20260928 05:17:53.050310 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:53.009849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52262 (local address 127.0.97.61:40533)
0928 05:17:53.024645 (+ 14796us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:53.024650 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:53.035333 (+ 10683us) server_negotiation.cc:407] Connection header received
0928 05:17:53.035565 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:53.035570 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:53.035656 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:53.035718 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:53.037858 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:53.038779 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:53.040304 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:53.040971 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:53.041063 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:53.044517 (+ 3454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:53.044523 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:53.044528 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:53.044560 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:53.047602 (+ 3042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:53.048509 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:53.048514 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:53.048517 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:53.048656 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:53.049037 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:53.049042 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:53.049046 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:53.049302 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:53.049531 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:53.049832 (+ 301us) server_negotiation.cc:299] Negotiation successful
0928 05:17:53.050019 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14693,"thread_start_us":24904,"threads_started":1}
I20260928 05:17:53.050171 6539 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60578:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 05:17:53.057271 6539 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 05:17:53.089761 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:53.074051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43633 (local address 127.0.0.1:32920)
0928 05:17:53.082433 (+ 8382us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:53.082449 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:53.082614 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:53.083362 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:53.083364 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:53.083382 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:17:53.083632 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:53.083638 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:53.084870 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:53.084876 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:53.086947 (+ 2071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:53.086962 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:53.088978 (+ 2016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:53.088985 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:53.089165 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:53.089197 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 05:17:53.089284 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 05:17:53.089397 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8242,"thread_start_us":253,"threads_started":1}
I20260928 05:17:53.089805 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:53.074255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32920 (local address 127.0.97.1:43633)
0928 05:17:53.077177 (+ 2922us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:53.077184 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:53.082525 (+ 5341us) server_negotiation.cc:407] Connection header received
0928 05:17:53.082794 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:53.082801 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:53.083034 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:53.083185 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:17:53.083788 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:53.084730 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:53.087093 (+ 2363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:53.088443 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:53.088557 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:53.088760 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:53.089355 (+ 595us) server_negotiation.cc:299] Negotiation successful
0928 05:17:53.089520 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2786,"thread_start_us":2694,"threads_started":1}
I20260928 05:17:53.090426 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:53.073312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46223 (local address 127.0.0.1:40574)
0928 05:17:53.074548 (+ 1236us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:53.074566 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:53.074650 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:53.081773 (+ 7123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:53.081778 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:53.081802 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:17:53.082164 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:53.082176 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:53.085813 (+ 3637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:53.085818 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:53.087873 (+ 2055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:53.087884 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:53.089670 (+ 1786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:53.089678 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:53.089863 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:53.089892 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 05:17:53.089979 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 05:17:53.090114 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1133,"thread_start_us":8849,"threads_started":1}
I20260928 05:17:53.091037 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:53.074160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40574 (local address 127.0.97.3:46223)
0928 05:17:53.081269 (+ 7109us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:53.081275 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:53.081300 (+ 25us) server_negotiation.cc:407] Connection header received
0928 05:17:53.081374 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:53.081379 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:53.081607 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:53.081780 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:17:53.082347 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:53.085680 (+ 3333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:53.088026 (+ 2346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:53.089244 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:53.089356 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:53.089527 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:53.090648 (+ 1121us) server_negotiation.cc:299] Negotiation successful
0928 05:17:53.090794 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6986,"thread_start_us":6839,"threads_started":1}
I20260928 05:17:53.093515 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:53.082476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33089 (local address 127.0.0.1:47864)
0928 05:17:53.086756 (+ 4280us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:53.086769 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:53.086884 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:53.087551 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:53.087556 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:53.087578 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:17:53.087973 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:53.087983 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:53.089604 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:53.089608 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:53.091680 (+ 2072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:53.091693 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:53.092790 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:53.092796 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:53.093009 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:53.093039 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 05:17:53.093122 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 05:17:53.093227 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4176,"thread_start_us":4043,"threads_started":1}
I20260928 05:17:53.093226 7438 tablet_service.cc:1467] Processing CreateTablet for tablet 8768c69fee984d698739cb35b5caaa1e (DEFAULT_TABLE table=test-table [id=fbdfe35d5cc240cfa2ff52141bd92043]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:17:53.094677 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:53.082692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47864 (local address 127.0.97.2:33089)
0928 05:17:53.083138 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:53.083145 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:53.086862 (+ 3717us) server_negotiation.cc:407] Connection header received
0928 05:17:53.087012 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:53.087019 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:53.087216 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:53.087377 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:17:53.088141 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:53.089452 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:53.091818 (+ 2366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:53.092612 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:53.092706 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:53.092848 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:53.093796 (+ 948us) server_negotiation.cc:299] Negotiation successful
0928 05:17:53.093961 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":302,"threads_started":1}
I20260928 05:17:53.094751 7438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8768c69fee984d698739cb35b5caaa1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:17:53.095769 7722 tablet_service.cc:1467] Processing CreateTablet for tablet 8768c69fee984d698739cb35b5caaa1e (DEFAULT_TABLE table=test-table [id=fbdfe35d5cc240cfa2ff52141bd92043]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:17:53.097050 7722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8768c69fee984d698739cb35b5caaa1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:17:53.097765 7582 tablet_service.cc:1467] Processing CreateTablet for tablet 8768c69fee984d698739cb35b5caaa1e (DEFAULT_TABLE table=test-table [id=fbdfe35d5cc240cfa2ff52141bd92043]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:17:53.098938 7582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8768c69fee984d698739cb35b5caaa1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:17:53.106842 7821 tablet_bootstrap.cc:492] T 8768c69fee984d698739cb35b5caaa1e P dafeb19659594f5abdc1fc79acdfb9df: Bootstrap starting.
I20260928 05:17:53.109541 7821 tablet_bootstrap.cc:654] T 8768c69fee984d698739cb35b5caaa1e P dafeb19659594f5abdc1fc79acdfb9df: Neither blocks nor log segments found. Creating new log.
I20260928 05:17:53.110565 7821 log.cc:824] T 8768c69fee984d698739cb35b5caaa1e P dafeb19659594f5abdc1fc79acdfb9df: Log is configured to *not* fsync() on all Append() calls
I20260928 05:17:53.111873 7822 tablet_bootstrap.cc:492] T 8768c69fee984d698739cb35b5caaa1e P a2d63277ac00458a83c7cb684be08738: Bootstrap starting.
I20260928 05:17:53.113263 7824 tablet_bootstrap.cc:492] T 8768c69fee984d698739cb35b5caaa1e P 176eae27f2854c7a84d8a0e5c342d0a9: Bootstrap starting.
I20260928 05:17:53.114672 7821 tablet_bootstrap.cc:492] T 8768c69fee984d698739cb35b5caaa1e P dafeb19659594f5abdc1fc79acdfb9df: No bootstrap required, opened a new log
I20260928 05:17:53.114979 7821 ts_tablet_manager.cc:1397] T 8768c69fee984d698739cb35b5caaa1e P dafeb19659594f5abdc1fc79acdfb9df: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260928 05:17:53.115506 7822 tablet_bootstrap.cc:654] T 8768c69fee984d698739cb35b5caaa1e P a2d63277ac00458a83c7cb684be08738: Neither blocks nor log segments found. Creating new log.
I20260928 05:17:53.115906 7824 tablet_bootstrap.cc:654] T 8768c69fee984d698739cb35b5caaa1e P 176eae27f2854c7a84d8a0e5c342d0a9: Neither blocks nor log segments found. Creating new log.
I20260928 05:17:53.116585 7822 log.cc:824] T 8768c69fee984d698739cb35b5caaa1e P a2d63277ac00458a83c7cb684be08738: Log is configured to *not* fsync() on all Append() calls
I20260928 05:17:53.116748 7824 log.cc:824] T 8768c69fee984d698739cb35b5caaa1e P 176eae27f2854c7a84d8a0e5c342d0a9: Log is configured to *not* fsync() on all Append() calls
I20260928 05:17:53.119922 7822 tablet_bootstrap.cc:492] T 8768c69fee984d698739cb35b5caaa1e P a2d63277ac00458a83c7cb684be08738: No bootstrap required, opened a new log
I20260928 05:17:53.120165 7822 ts_tablet_manager.cc:1397] T 8768c69fee984d698739cb35b5caaa1e P a2d63277ac00458a83c7cb684be08738: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260928 05:17:53.124285 7821 raft_consensus.cc:348] T 8768c69fee984d698739cb35b5caaa1e P dafeb19659594f5abdc1fc79acdfb9df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2d63277ac00458a83c7cb684be08738" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46223 } } peers { permanent_uuid: "176eae27f2854c7a84d8a0e5c342d0a9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33089 } } peers { permanent_uuid: "dafeb19659594f5abdc1fc79acdfb9df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43633 } }
I20260928 05:17:53.124670 7821 raft_consensus.cc:374] T 8768c69fee984d698739cb35b5caaa1e P dafeb19659594f5abdc1fc79acdfb9df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:17:53.124755 7821 raft_consensus.cc:729] T 8768c69fee984d698739cb35b5caaa1e P dafeb19659594f5abdc1fc79acdfb9df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dafeb19659594f5abdc1fc79acdfb9df, State: Initialized, Role: FOLLOWER
I20260928 05:17:53.125387 7821 consensus_queue.cc:260] T 8768c69fee984d698739cb35b5caaa1e P dafeb19659594f5abdc1fc79acdfb9df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2d63277ac00458a83c7cb684be08738" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46223 } } peers { permanent_uuid: "176eae27f2854c7a84d8a0e5c342d0a9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33089 } } peers { permanent_uuid: "dafeb19659594f5abdc1fc79acdfb9df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43633 } }
I20260928 05:17:53.127914 7824 tablet_bootstrap.cc:492] T 8768c69fee984d698739cb35b5caaa1e P 176eae27f2854c7a84d8a0e5c342d0a9: No bootstrap required, opened a new log
I20260928 05:17:53.128142 7824 ts_tablet_manager.cc:1397] T 8768c69fee984d698739cb35b5caaa1e P 176eae27f2854c7a84d8a0e5c342d0a9: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.010s
I20260928 05:17:53.130613 7822 raft_consensus.cc:348] T 8768c69fee984d698739cb35b5caaa1e P a2d63277ac00458a83c7cb684be08738 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2d63277ac00458a83c7cb684be08738" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46223 } } peers { permanent_uuid: "176eae27f2854c7a84d8a0e5c342d0a9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33089 } } peers { permanent_uuid: "dafeb19659594f5abdc1fc79acdfb9df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43633 } }
I20260928 05:17:53.130980 7822 raft_consensus.cc:374] T 8768c69fee984d698739cb35b5caaa1e P a2d63277ac00458a83c7cb684be08738 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:17:53.131070 7822 raft_consensus.cc:729] T 8768c69fee984d698739cb35b5caaa1e P a2d63277ac00458a83c7cb684be08738 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2d63277ac00458a83c7cb684be08738, State: Initialized, Role: FOLLOWER
I20260928 05:17:53.131690 7822 consensus_queue.cc:260] T 8768c69fee984d698739cb35b5caaa1e P a2d63277ac00458a83c7cb684be08738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2d63277ac00458a83c7cb684be08738" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46223 } } peers { permanent_uuid: "176eae27f2854c7a84d8a0e5c342d0a9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33089 } } peers { permanent_uuid: "dafeb19659594f5abdc1fc79acdfb9df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43633 } }
I20260928 05:17:53.132463 7821 ts_tablet_manager.cc:1428] T 8768c69fee984d698739cb35b5caaa1e P dafeb19659594f5abdc1fc79acdfb9df: Time spent starting tablet: real 0.017s user 0.011s sys 0.000s
I20260928 05:17:53.133229 7822 ts_tablet_manager.cc:1428] T 8768c69fee984d698739cb35b5caaa1e P a2d63277ac00458a83c7cb684be08738: Time spent starting tablet: real 0.013s user 0.010s sys 0.004s
I20260928 05:17:53.133467 7788 heartbeater.cc:503] Master 127.0.97.62:38331 was elected leader, sending a full tablet report...
I20260928 05:17:53.137401 7824 raft_consensus.cc:348] T 8768c69fee984d698739cb35b5caaa1e P 176eae27f2854c7a84d8a0e5c342d0a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2d63277ac00458a83c7cb684be08738" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46223 } } peers { permanent_uuid: "176eae27f2854c7a84d8a0e5c342d0a9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33089 } } peers { permanent_uuid: "dafeb19659594f5abdc1fc79acdfb9df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43633 } }
I20260928 05:17:53.137784 7824 raft_consensus.cc:374] T 8768c69fee984d698739cb35b5caaa1e P 176eae27f2854c7a84d8a0e5c342d0a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:17:53.137866 7824 raft_consensus.cc:729] T 8768c69fee984d698739cb35b5caaa1e P 176eae27f2854c7a84d8a0e5c342d0a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 176eae27f2854c7a84d8a0e5c342d0a9, State: Initialized, Role: FOLLOWER
I20260928 05:17:53.138402 7824 consensus_queue.cc:260] T 8768c69fee984d698739cb35b5caaa1e P 176eae27f2854c7a84d8a0e5c342d0a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2d63277ac00458a83c7cb684be08738" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46223 } } peers { permanent_uuid: "176eae27f2854c7a84d8a0e5c342d0a9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33089 } } peers { permanent_uuid: "dafeb19659594f5abdc1fc79acdfb9df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43633 } }
I20260928 05:17:53.139241 7648 heartbeater.cc:503] Master 127.0.97.62:38331 was elected leader, sending a full tablet report...
I20260928 05:17:53.139762 7824 ts_tablet_manager.cc:1428] T 8768c69fee984d698739cb35b5caaa1e P 176eae27f2854c7a84d8a0e5c342d0a9: Time spent starting tablet: real 0.012s user 0.011s sys 0.001s
W20260928 05:17:53.154596 7650 tablet.cc:2369] T 8768c69fee984d698739cb35b5caaa1e P 176eae27f2854c7a84d8a0e5c342d0a9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 05:17:53.240386 7790 tablet.cc:2369] T 8768c69fee984d698739cb35b5caaa1e P a2d63277ac00458a83c7cb684be08738: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 05:17:53.280689 7829 raft_consensus.cc:482] T 8768c69fee984d698739cb35b5caaa1e P 176eae27f2854c7a84d8a0e5c342d0a9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:17:53.280983 7829 raft_consensus.cc:504] T 8768c69fee984d698739cb35b5caaa1e P 176eae27f2854c7a84d8a0e5c342d0a9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2d63277ac00458a83c7cb684be08738" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46223 } } peers { permanent_uuid: "176eae27f2854c7a84d8a0e5c342d0a9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33089 } } peers { permanent_uuid: "dafeb19659594f5abdc1fc79acdfb9df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43633 } }
I20260928 05:17:53.282542 7829 leader_election.cc:288] T 8768c69fee984d698739cb35b5caaa1e P 176eae27f2854c7a84d8a0e5c342d0a9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2d63277ac00458a83c7cb684be08738 (127.0.97.3:46223), dafeb19659594f5abdc1fc79acdfb9df (127.0.97.1:43633)
I20260928 05:17:53.288424 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:53.282813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34611 (local address 127.0.97.3:46223)
0928 05:17:53.282981 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:53.282986 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:53.283699 (+ 713us) server_negotiation.cc:407] Connection header received
0928 05:17:53.283864 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:53.283868 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:53.283922 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:53.284002 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:17:53.284433 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:53.285194 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:53.286927 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:53.287634 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:53.287739 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:53.287833 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:53.288081 (+ 248us) server_negotiation.cc:299] Negotiation successful
0928 05:17:53.288193 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 05:17:53.288708 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:53.282936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48693 (local address 127.0.97.1:43633)
0928 05:17:53.283111 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:53.283117 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:53.283735 (+ 618us) server_negotiation.cc:407] Connection header received
0928 05:17:53.283899 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:53.283903 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:53.283962 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:53.284034 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:17:53.284750 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:53.285524 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:53.287176 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:53.287922 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:53.288030 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:53.288127 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:53.288379 (+ 252us) server_negotiation.cc:299] Negotiation successful
0928 05:17:53.288498 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 05:17:53.289036 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:53.282717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46223 (local address 127.0.97.2:34611)
0928 05:17:53.283596 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:53.283611 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:53.283742 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:53.284017 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:53.284021 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:53.284041 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:17:53.284304 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:53.284312 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:53.285330 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:53.285334 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:53.286785 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:53.286796 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:53.287805 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:53.287811 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:53.287939 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:53.287960 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 05:17:53.288040 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 05:17:53.288160 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":781,"thread_start_us":564,"threads_started":1}
I20260928 05:17:53.289283 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:53.282742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43633 (local address 127.0.97.2:48693)
0928 05:17:53.283645 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:53.283659 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:53.283774 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:53.284364 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:53.284367 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:53.284385 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:17:53.284596 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:53.284602 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:53.285655 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:53.285659 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:53.287045 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:53.287057 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:53.288112 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:53.288119 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:53.288243 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:53.288268 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 05:17:53.288342 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 05:17:53.288450 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":799,"thread_start_us":584,"threads_started":1}
I20260928 05:17:53.289901 7742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8768c69fee984d698739cb35b5caaa1e" candidate_uuid: "176eae27f2854c7a84d8a0e5c342d0a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2d63277ac00458a83c7cb684be08738" is_pre_election: true
I20260928 05:17:53.290066 7458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8768c69fee984d698739cb35b5caaa1e" candidate_uuid: "176eae27f2854c7a84d8a0e5c342d0a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dafeb19659594f5abdc1fc79acdfb9df" is_pre_election: true
I20260928 05:17:53.290472 7458 raft_consensus.cc:2454] T 8768c69fee984d698739cb35b5caaa1e P dafeb19659594f5abdc1fc79acdfb9df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 176eae27f2854c7a84d8a0e5c342d0a9 in term 0.
I20260928 05:17:53.290465 7742 raft_consensus.cc:2454] T 8768c69fee984d698739cb35b5caaa1e P a2d63277ac00458a83c7cb684be08738 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 176eae27f2854c7a84d8a0e5c342d0a9 in term 0.
I20260928 05:17:53.291049 7538 leader_election.cc:302] T 8768c69fee984d698739cb35b5caaa1e P 176eae27f2854c7a84d8a0e5c342d0a9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 176eae27f2854c7a84d8a0e5c342d0a9, dafeb19659594f5abdc1fc79acdfb9df; no voters:
I20260928 05:17:53.291379 7829 raft_consensus.cc:2781] T 8768c69fee984d698739cb35b5caaa1e P 176eae27f2854c7a84d8a0e5c342d0a9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 05:17:53.291520 7829 raft_consensus.cc:482] T 8768c69fee984d698739cb35b5caaa1e P 176eae27f2854c7a84d8a0e5c342d0a9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 05:17:53.291620 7829 raft_consensus.cc:3037] T 8768c69fee984d698739cb35b5caaa1e P 176eae27f2854c7a84d8a0e5c342d0a9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:17:53.293649 7829 raft_consensus.cc:504] T 8768c69fee984d698739cb35b5caaa1e P 176eae27f2854c7a84d8a0e5c342d0a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2d63277ac00458a83c7cb684be08738" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46223 } } peers { permanent_uuid: "176eae27f2854c7a84d8a0e5c342d0a9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33089 } } peers { permanent_uuid: "dafeb19659594f5abdc1fc79acdfb9df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43633 } }
I20260928 05:17:53.294130 7829 leader_election.cc:288] T 8768c69fee984d698739cb35b5caaa1e P 176eae27f2854c7a84d8a0e5c342d0a9 [CANDIDATE]: Term 1 election: Requested vote from peers a2d63277ac00458a83c7cb684be08738 (127.0.97.3:46223), dafeb19659594f5abdc1fc79acdfb9df (127.0.97.1:43633)
I20260928 05:17:53.294539 7458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8768c69fee984d698739cb35b5caaa1e" candidate_uuid: "176eae27f2854c7a84d8a0e5c342d0a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dafeb19659594f5abdc1fc79acdfb9df"
I20260928 05:17:53.294546 7742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8768c69fee984d698739cb35b5caaa1e" candidate_uuid: "176eae27f2854c7a84d8a0e5c342d0a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2d63277ac00458a83c7cb684be08738"
I20260928 05:17:53.294694 7458 raft_consensus.cc:3037] T 8768c69fee984d698739cb35b5caaa1e P dafeb19659594f5abdc1fc79acdfb9df [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:17:53.294716 7742 raft_consensus.cc:3037] T 8768c69fee984d698739cb35b5caaa1e P a2d63277ac00458a83c7cb684be08738 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:17:53.296671 7458 raft_consensus.cc:2454] T 8768c69fee984d698739cb35b5caaa1e P dafeb19659594f5abdc1fc79acdfb9df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 176eae27f2854c7a84d8a0e5c342d0a9 in term 1.
I20260928 05:17:53.296977 7742 raft_consensus.cc:2454] T 8768c69fee984d698739cb35b5caaa1e P a2d63277ac00458a83c7cb684be08738 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 176eae27f2854c7a84d8a0e5c342d0a9 in term 1.
I20260928 05:17:53.297021 7538 leader_election.cc:302] T 8768c69fee984d698739cb35b5caaa1e P 176eae27f2854c7a84d8a0e5c342d0a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 176eae27f2854c7a84d8a0e5c342d0a9, dafeb19659594f5abdc1fc79acdfb9df; no voters:
I20260928 05:17:53.297250 7829 raft_consensus.cc:2781] T 8768c69fee984d698739cb35b5caaa1e P 176eae27f2854c7a84d8a0e5c342d0a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:17:53.297694 7829 raft_consensus.cc:686] T 8768c69fee984d698739cb35b5caaa1e P 176eae27f2854c7a84d8a0e5c342d0a9 [term 1 LEADER]: Becoming Leader. State: Replica: 176eae27f2854c7a84d8a0e5c342d0a9, State: Running, Role: LEADER
I20260928 05:17:53.298198 7829 consensus_queue.cc:237] T 8768c69fee984d698739cb35b5caaa1e P 176eae27f2854c7a84d8a0e5c342d0a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2d63277ac00458a83c7cb684be08738" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46223 } } peers { permanent_uuid: "176eae27f2854c7a84d8a0e5c342d0a9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33089 } } peers { permanent_uuid: "dafeb19659594f5abdc1fc79acdfb9df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43633 } }
I20260928 05:17:53.303122 6538 catalog_manager.cc:5495] T 8768c69fee984d698739cb35b5caaa1e P 176eae27f2854c7a84d8a0e5c342d0a9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 176eae27f2854c7a84d8a0e5c342d0a9 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "176eae27f2854c7a84d8a0e5c342d0a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2d63277ac00458a83c7cb684be08738" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "176eae27f2854c7a84d8a0e5c342d0a9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dafeb19659594f5abdc1fc79acdfb9df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43633 } health_report { overall_health: UNKNOWN } } }
I20260928 05:17:53.319861 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:53.315229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47876 (local address 127.0.97.2:33089)
0928 05:17:53.315403 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:53.315408 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:53.315424 (+ 16us) server_negotiation.cc:407] Connection header received
0928 05:17:53.315508 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:53.315512 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:53.315576 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:53.315692 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:17:53.316025 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:53.316685 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:53.317971 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:53.318343 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:53.318449 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:53.318866 (+ 417us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:17:53.319330 (+ 464us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:17:53.319459 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:53.319559 (+ 100us) server_negotiation.cc:299] Negotiation successful
0928 05:17:53.319630 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 05:17:53.327010 7458 raft_consensus.cc:1261] T 8768c69fee984d698739cb35b5caaa1e P dafeb19659594f5abdc1fc79acdfb9df [term 1 FOLLOWER]: Refusing update from remote peer 176eae27f2854c7a84d8a0e5c342d0a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:17:53.327013 7742 raft_consensus.cc:1261] T 8768c69fee984d698739cb35b5caaa1e P a2d63277ac00458a83c7cb684be08738 [term 1 FOLLOWER]: Refusing update from remote peer 176eae27f2854c7a84d8a0e5c342d0a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:17:53.328212 7832 consensus_queue.cc:1035] T 8768c69fee984d698739cb35b5caaa1e P 176eae27f2854c7a84d8a0e5c342d0a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dafeb19659594f5abdc1fc79acdfb9df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260928 05:17:53.328516 7829 consensus_queue.cc:1035] T 8768c69fee984d698739cb35b5caaa1e P 176eae27f2854c7a84d8a0e5c342d0a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2d63277ac00458a83c7cb684be08738" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:17:53.337080 7835 mvcc.cc:205] Tried to move back new op lower bound from 7334185669939068928 to 7334185669832798208. Current Snapshot: MvccSnapshot[applied={T|T < 7334185669939068928}]
I20260928 05:17:53.337604 7836 mvcc.cc:205] Tried to move back new op lower bound from 7334185669939068928 to 7334185669832798208. Current Snapshot: MvccSnapshot[applied={T|T < 7334185669939068928}]
I20260928 05:17:53.356715 6538 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60578:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:17:53.356976 6538 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60578:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:17:53.362277 6538 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 401619c49c134f59a36d3d960b3589ee: Sending DeleteTablet for 3 replicas of tablet 8768c69fee984d698739cb35b5caaa1e
I20260928 05:17:53.363903 7438 tablet_service.cc:1514] Processing DeleteTablet for tablet 8768c69fee984d698739cb35b5caaa1e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:17:53 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:32920
I20260928 05:17:53.364007 7722 tablet_service.cc:1514] Processing DeleteTablet for tablet 8768c69fee984d698739cb35b5caaa1e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:17:53 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40574
I20260928 05:17:53.364960 7582 tablet_service.cc:1514] Processing DeleteTablet for tablet 8768c69fee984d698739cb35b5caaa1e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:17:53 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47864
I20260928 05:17:53.375434 7847 tablet_replica.cc:330] T 8768c69fee984d698739cb35b5caaa1e P a2d63277ac00458a83c7cb684be08738: stopping tablet replica
I20260928 05:17:53.375864 7847 raft_consensus.cc:2229] T 8768c69fee984d698739cb35b5caaa1e P a2d63277ac00458a83c7cb684be08738 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:17:53.376366 7847 raft_consensus.cc:2258] T 8768c69fee984d698739cb35b5caaa1e P a2d63277ac00458a83c7cb684be08738 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:17:53.377311 7846 tablet_replica.cc:330] T 8768c69fee984d698739cb35b5caaa1e P dafeb19659594f5abdc1fc79acdfb9df: stopping tablet replica
I20260928 05:17:53.377573 7846 raft_consensus.cc:2229] T 8768c69fee984d698739cb35b5caaa1e P dafeb19659594f5abdc1fc79acdfb9df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:17:53.377889 7846 raft_consensus.cc:2258] T 8768c69fee984d698739cb35b5caaa1e P dafeb19659594f5abdc1fc79acdfb9df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:17:53.379132 7847 ts_tablet_manager.cc:1905] T 8768c69fee984d698739cb35b5caaa1e P a2d63277ac00458a83c7cb684be08738: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:17:53.383616 7847 ts_tablet_manager.cc:1918] T 8768c69fee984d698739cb35b5caaa1e P a2d63277ac00458a83c7cb684be08738: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:17:53.383750 7847 log.cc:1196] T 8768c69fee984d698739cb35b5caaa1e P a2d63277ac00458a83c7cb684be08738: Deleting WAL directory at /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/ts-2/wal/wals/8768c69fee984d698739cb35b5caaa1e
I20260928 05:17:53.384248 7847 ts_tablet_manager.cc:1939] T 8768c69fee984d698739cb35b5caaa1e P a2d63277ac00458a83c7cb684be08738: Deleting consensus metadata
I20260928 05:17:53.386001 7846 ts_tablet_manager.cc:1905] T 8768c69fee984d698739cb35b5caaa1e P dafeb19659594f5abdc1fc79acdfb9df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:17:53.386020 6523 catalog_manager.cc:4841] TS a2d63277ac00458a83c7cb684be08738 (127.0.97.3:46223): tablet 8768c69fee984d698739cb35b5caaa1e (table test-table [id=fbdfe35d5cc240cfa2ff52141bd92043]) successfully deleted
I20260928 05:17:53.389986 7848 tablet_replica.cc:330] T 8768c69fee984d698739cb35b5caaa1e P 176eae27f2854c7a84d8a0e5c342d0a9: stopping tablet replica
I20260928 05:17:53.390390 7848 raft_consensus.cc:2229] T 8768c69fee984d698739cb35b5caaa1e P 176eae27f2854c7a84d8a0e5c342d0a9 [term 1 LEADER]: Raft consensus shutting down.
I20260928 05:17:53.390825 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 6507
I20260928 05:17:53.391000 7848 raft_consensus.cc:2258] T 8768c69fee984d698739cb35b5caaa1e P 176eae27f2854c7a84d8a0e5c342d0a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:17:53.391381 7846 ts_tablet_manager.cc:1918] T 8768c69fee984d698739cb35b5caaa1e P dafeb19659594f5abdc1fc79acdfb9df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:17:53.391595 7846 log.cc:1196] T 8768c69fee984d698739cb35b5caaa1e P dafeb19659594f5abdc1fc79acdfb9df: Deleting WAL directory at /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/ts-0/wal/wals/8768c69fee984d698739cb35b5caaa1e
I20260928 05:17:53.391809 6569 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:17:53.392256 7846 ts_tablet_manager.cc:1939] T 8768c69fee984d698739cb35b5caaa1e P dafeb19659594f5abdc1fc79acdfb9df: Deleting consensus metadata
I20260928 05:17:53.395437 7848 ts_tablet_manager.cc:1905] T 8768c69fee984d698739cb35b5caaa1e P 176eae27f2854c7a84d8a0e5c342d0a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:17:53.402868 7848 ts_tablet_manager.cc:1918] T 8768c69fee984d698739cb35b5caaa1e P 176eae27f2854c7a84d8a0e5c342d0a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:17:53.404500 7848 log.cc:1196] T 8768c69fee984d698739cb35b5caaa1e P 176eae27f2854c7a84d8a0e5c342d0a9: Deleting WAL directory at /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/ts-1/wal/wals/8768c69fee984d698739cb35b5caaa1e
I20260928 05:17:53.405288 7848 ts_tablet_manager.cc:1939] T 8768c69fee984d698739cb35b5caaa1e P 176eae27f2854c7a84d8a0e5c342d0a9: Deleting consensus metadata
I20260928 05:17:53.412021 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:53.367869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52274 (local address 127.0.97.61:40533)
0928 05:17:53.368027 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:53.368033 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:53.389636 (+ 21603us) server_negotiation.cc:407] Connection header received
0928 05:17:53.391557 (+ 1921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:53.391563 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:53.391649 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:53.391741 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:53.394597 (+ 2856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:53.395659 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:53.397646 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:53.398227 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:53.398319 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:53.405291 (+ 6972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:53.405299 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:53.405305 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:53.405348 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:53.408915 (+ 3567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:53.409758 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:53.409764 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:53.409768 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:53.409899 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:53.410296 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:53.410301 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:53.410304 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:53.410700 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:53.410934 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:53.411462 (+ 528us) server_negotiation.cc:299] Negotiation successful
0928 05:17:53.411683 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 05:17:53.429322 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:53.389783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51882 (local address 127.0.97.60:39701)
0928 05:17:53.389966 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:53.389971 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:53.414423 (+ 24452us) server_negotiation.cc:407] Connection header received
0928 05:17:53.414664 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:53.414669 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:53.414749 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:53.414817 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:53.416482 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:53.417537 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:53.419485 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:53.420116 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:53.420205 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:53.423333 (+ 3128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:53.423340 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:53.423345 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:53.423382 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:53.426473 (+ 3091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:53.427201 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:53.427206 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:53.427208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:53.427333 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:53.427756 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:53.427761 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:53.427764 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:53.428065 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:53.428278 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:53.428818 (+ 540us) server_negotiation.cc:299] Negotiation successful
0928 05:17:53.429029 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 05:17:53.531379 6523 catalog_manager.cc:4841] TS 176eae27f2854c7a84d8a0e5c342d0a9 (127.0.97.2:33089): tablet 8768c69fee984d698739cb35b5caaa1e (table test-table [id=fbdfe35d5cc240cfa2ff52141bd92043]) successfully deleted
I20260928 05:17:53.533237 6526 catalog_manager.cc:4841] TS dafeb19659594f5abdc1fc79acdfb9df (127.0.97.1:43633): tablet 8768c69fee984d698739cb35b5caaa1e (table test-table [id=fbdfe35d5cc240cfa2ff52141bd92043]) successfully deleted
W20260928 05:17:53.533465 6526 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.97.1:43633, user_credentials={real_user=slave}} blocked reactor thread for 139547us
I20260928 05:17:53.533896 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 6507
I20260928 05:17:53.577257 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:53.555881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52284 (local address 127.0.97.61:40533)
0928 05:17:53.556060 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:53.556067 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:53.559356 (+ 3289us) server_negotiation.cc:407] Connection header received
0928 05:17:53.560240 (+ 884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:53.560246 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:53.560328 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:53.560401 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:53.561859 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:53.562887 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:53.564327 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:53.564748 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:53.564883 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:53.571705 (+ 6822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:53.571712 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:53.571716 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:53.571754 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:53.574624 (+ 2870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:53.575304 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:53.575310 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:53.575313 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:53.575444 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:53.575800 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:53.575805 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:53.575807 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:53.576094 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:53.576313 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:53.576685 (+ 372us) server_negotiation.cc:299] Negotiation successful
0928 05:17:53.576921 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 05:17:53.577301 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:53.555146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51892 (local address 127.0.97.60:39701)
0928 05:17:53.555723 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:53.555730 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:53.560122 (+ 4392us) server_negotiation.cc:407] Connection header received
0928 05:17:53.561967 (+ 1845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:53.561972 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:53.562036 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:53.562113 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:53.563644 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:53.564701 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:53.566147 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:53.566710 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:53.566830 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:53.571041 (+ 4211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:53.571045 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:53.571048 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:53.571077 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:53.574061 (+ 2984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:53.574818 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:53.574823 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:53.574826 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:53.574960 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:53.575332 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:53.575336 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:53.575339 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:53.575647 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:53.575897 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:53.576775 (+ 878us) server_negotiation.cc:299] Negotiation successful
0928 05:17:53.576978 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450}
I20260928 05:17:54.345014 7858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:54.344217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38331 (local address 127.0.97.1:46921)
0928 05:17:54.344671 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:54.344806 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":204,"threads_started":1}
W20260928 05:17:54.345368 7504 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:38331 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:38331: connect: Connection refused (error 111)
I20260928 05:17:54.349431 7859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:54.348718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38331 (local address 127.0.97.3:54365)
0928 05:17:54.349139 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:54.349264 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":230,"threads_started":1}
W20260928 05:17:54.349763 7788 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:38331 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:38331: connect: Connection refused (error 111)
I20260928 05:17:54.353962 7860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:54.353263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38331 (local address 127.0.97.2:34319)
0928 05:17:54.353669 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:54.353806 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":221,"threads_started":1}
W20260928 05:17:54.354316 7648 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:38331 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:38331: connect: Connection refused (error 111)
I20260928 05:17:54.862644 7861 raft_consensus.cc:482] T 00000000000000000000000000000000 P 599975ad9f374b9cba8b0d1e09b84822 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 401619c49c134f59a36d3d960b3589ee)
I20260928 05:17:54.862890 7861 raft_consensus.cc:504] T 00000000000000000000000000000000 P 599975ad9f374b9cba8b0d1e09b84822 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "401619c49c134f59a36d3d960b3589ee" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38331 } } peers { permanent_uuid: "109bdf09ab804bf1b13ca35cce2e6bcf" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40533 } attrs { promote: false } } peers { permanent_uuid: "599975ad9f374b9cba8b0d1e09b84822" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39701 } attrs { promote: false } }
I20260928 05:17:54.864311 7861 leader_election.cc:288] T 00000000000000000000000000000000 P 599975ad9f374b9cba8b0d1e09b84822 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 401619c49c134f59a36d3d960b3589ee (127.0.97.62:38331), 109bdf09ab804bf1b13ca35cce2e6bcf (127.0.97.61:40533)
I20260928 05:17:54.864355 7862 raft_consensus.cc:482] T 00000000000000000000000000000000 P 109bdf09ab804bf1b13ca35cce2e6bcf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 401619c49c134f59a36d3d960b3589ee)
I20260928 05:17:54.864588 7862 raft_consensus.cc:504] T 00000000000000000000000000000000 P 109bdf09ab804bf1b13ca35cce2e6bcf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "401619c49c134f59a36d3d960b3589ee" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38331 } } peers { permanent_uuid: "109bdf09ab804bf1b13ca35cce2e6bcf" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40533 } attrs { promote: false } } peers { permanent_uuid: "599975ad9f374b9cba8b0d1e09b84822" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39701 } attrs { promote: false } }
I20260928 05:17:54.865903 7863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:54.864396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38331 (local address 127.0.0.1:60990)
0928 05:17:54.865064 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:54.865673 (+ 609us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":383,"threads_started":1}
W20260928 05:17:54.866196 7309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38331: connect: Connection refused (error 111)
I20260928 05:17:54.866479 7866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:54.865604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38331 (local address 127.0.0.1:60992)
0928 05:17:54.866061 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:54.866197 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":220,"threads_started":1}
W20260928 05:17:54.867055 7120 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38331: connect: Connection refused (error 111)
I20260928 05:17:54.872485 7862 leader_election.cc:288] T 00000000000000000000000000000000 P 109bdf09ab804bf1b13ca35cce2e6bcf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 401619c49c134f59a36d3d960b3589ee (127.0.97.62:38331), 599975ad9f374b9cba8b0d1e09b84822 (127.0.97.60:39701)
I20260928 05:17:54.873179 7864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:54.864524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40533 (local address 127.0.0.1:52292)
0928 05:17:54.865362 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:54.865379 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:54.865512 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:54.866161 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:54.866166 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:54.866198 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:17:54.866506 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:54.866535 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:54.868341 (+ 1806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:54.868348 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:54.870516 (+ 2168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:54.870532 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:54.872380 (+ 1848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:54.872386 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:54.872579 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:54.872613 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 05:17:54.872679 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 05:17:54.872799 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":711,"thread_start_us":538,"threads_started":1}
I20260928 05:17:54.874418 7865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:54.865315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52292 (local address 127.0.97.61:40533)
0928 05:17:54.865816 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:54.865821 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:54.865841 (+ 20us) server_negotiation.cc:407] Connection header received
0928 05:17:54.865922 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:54.865927 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:54.866005 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:54.866693 (+ 688us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:17:54.866923 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:54.868179 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:54.870820 (+ 2641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:54.872189 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:54.873710 (+ 1521us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:54.873809 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:54.873861 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 05:17:54.873996 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":214,"threads_started":1}
I20260928 05:17:54.875125 7146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "599975ad9f374b9cba8b0d1e09b84822" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "109bdf09ab804bf1b13ca35cce2e6bcf" is_pre_election: true
I20260928 05:17:54.875617 7146 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 109bdf09ab804bf1b13ca35cce2e6bcf [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 599975ad9f374b9cba8b0d1e09b84822 in term 1.
I20260928 05:17:54.876209 7310 leader_election.cc:302] T 00000000000000000000000000000000 P 599975ad9f374b9cba8b0d1e09b84822 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 109bdf09ab804bf1b13ca35cce2e6bcf, 599975ad9f374b9cba8b0d1e09b84822; no voters:
I20260928 05:17:54.876549 7861 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 599975ad9f374b9cba8b0d1e09b84822 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 05:17:54.876674 7861 raft_consensus.cc:482] T 00000000000000000000000000000000 P 599975ad9f374b9cba8b0d1e09b84822 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 401619c49c134f59a36d3d960b3589ee)
I20260928 05:17:54.876767 7861 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 599975ad9f374b9cba8b0d1e09b84822 [term 1 FOLLOWER]: Advancing to term 2
I20260928 05:17:54.878340 7864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:54.878004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38331 (local address 127.0.0.1:32790)
0928 05:17:54.878133 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:54.878206 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 05:17:54.878415 7867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:54.877489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38331 (local address 127.0.0.1:32776)
0928 05:17:54.878014 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:54.878161 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396}
W20260928 05:17:54.878594 7120 leader_election.cc:334] T 00000000000000000000000000000000 P 109bdf09ab804bf1b13ca35cce2e6bcf [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 401619c49c134f59a36d3d960b3589ee (127.0.97.62:38331): Network error: Client connection negotiation failed: client connection to 127.0.97.62:38331: connect: Connection refused (error 111)
W20260928 05:17:54.878731 7309 leader_election.cc:334] T 00000000000000000000000000000000 P 599975ad9f374b9cba8b0d1e09b84822 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 401619c49c134f59a36d3d960b3589ee (127.0.97.62:38331): Network error: Client connection negotiation failed: client connection to 127.0.97.62:38331: connect: Connection refused (error 111)
I20260928 05:17:54.879868 7861 raft_consensus.cc:504] T 00000000000000000000000000000000 P 599975ad9f374b9cba8b0d1e09b84822 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "401619c49c134f59a36d3d960b3589ee" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38331 } } peers { permanent_uuid: "109bdf09ab804bf1b13ca35cce2e6bcf" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40533 } attrs { promote: false } } peers { permanent_uuid: "599975ad9f374b9cba8b0d1e09b84822" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39701 } attrs { promote: false } }
I20260928 05:17:54.880455 7861 leader_election.cc:288] T 00000000000000000000000000000000 P 599975ad9f374b9cba8b0d1e09b84822 [CANDIDATE]: Term 2 election: Requested vote from peers 401619c49c134f59a36d3d960b3589ee (127.0.97.62:38331), 109bdf09ab804bf1b13ca35cce2e6bcf (127.0.97.61:40533)
I20260928 05:17:54.880897 7864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:54.880544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38331 (local address 127.0.0.1:32792)
0928 05:17:54.880651 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:54.880710 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 05:17:54.880908 7146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "599975ad9f374b9cba8b0d1e09b84822" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "109bdf09ab804bf1b13ca35cce2e6bcf"
I20260928 05:17:54.881115 7146 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 109bdf09ab804bf1b13ca35cce2e6bcf [term 1 FOLLOWER]: Advancing to term 2
I20260928 05:17:54.881845 7864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:54.881516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38331 (local address 127.0.0.1:32802)
0928 05:17:54.881659 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:54.881734 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260928 05:17:54.882042 7309 leader_election.cc:334] T 00000000000000000000000000000000 P 599975ad9f374b9cba8b0d1e09b84822 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 401619c49c134f59a36d3d960b3589ee (127.0.97.62:38331): Network error: Client connection negotiation failed: client connection to 127.0.97.62:38331: connect: Connection refused (error 111)
I20260928 05:17:54.883348 7146 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 109bdf09ab804bf1b13ca35cce2e6bcf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 599975ad9f374b9cba8b0d1e09b84822 in term 2.
I20260928 05:17:54.883649 7310 leader_election.cc:302] T 00000000000000000000000000000000 P 599975ad9f374b9cba8b0d1e09b84822 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 109bdf09ab804bf1b13ca35cce2e6bcf, 599975ad9f374b9cba8b0d1e09b84822; no voters: 401619c49c134f59a36d3d960b3589ee
I20260928 05:17:54.884019 7861 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 599975ad9f374b9cba8b0d1e09b84822 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 05:17:54.884603 7869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:54.866879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51900 (local address 127.0.97.60:39701)
0928 05:17:54.879268 (+ 12389us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:54.879272 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:54.879288 (+ 16us) server_negotiation.cc:407] Connection header received
0928 05:17:54.879347 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:54.879351 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:54.879406 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:54.879505 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:17:54.879940 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:54.880934 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:54.882859 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:54.883724 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:54.883835 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:54.883932 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:54.884254 (+ 322us) server_negotiation.cc:299] Negotiation successful
0928 05:17:54.884339 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12301,"thread_start_us":12189,"threads_started":1}
I20260928 05:17:54.884716 7866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:54.866379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39701 (local address 127.0.0.1:51900)
0928 05:17:54.866658 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:54.866673 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:54.866773 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:54.879526 (+ 12753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:54.879531 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:54.879556 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:17:54.879785 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:54.879793 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:54.881077 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:54.881082 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:54.882725 (+ 1643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:54.882737 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:54.883886 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:54.883892 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:54.884086 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:54.884141 (+ 55us) client_negotiation.cc:769] Sending connection context
0928 05:17:54.884238 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 05:17:54.884382 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":10701,"threads_started":1}
I20260928 05:17:54.885128 7861 raft_consensus.cc:686] T 00000000000000000000000000000000 P 599975ad9f374b9cba8b0d1e09b84822 [term 2 LEADER]: Becoming Leader. State: Replica: 599975ad9f374b9cba8b0d1e09b84822, State: Running, Role: LEADER
I20260928 05:17:54.885488 7335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "109bdf09ab804bf1b13ca35cce2e6bcf" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "599975ad9f374b9cba8b0d1e09b84822" is_pre_election: true
I20260928 05:17:54.885723 7861 consensus_queue.cc:237] T 00000000000000000000000000000000 P 599975ad9f374b9cba8b0d1e09b84822 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "401619c49c134f59a36d3d960b3589ee" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38331 } } peers { permanent_uuid: "109bdf09ab804bf1b13ca35cce2e6bcf" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40533 } attrs { promote: false } } peers { permanent_uuid: "599975ad9f374b9cba8b0d1e09b84822" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39701 } attrs { promote: false } }
I20260928 05:17:54.886510 7123 leader_election.cc:302] T 00000000000000000000000000000000 P 109bdf09ab804bf1b13ca35cce2e6bcf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 109bdf09ab804bf1b13ca35cce2e6bcf; no voters: 401619c49c134f59a36d3d960b3589ee, 599975ad9f374b9cba8b0d1e09b84822
I20260928 05:17:54.886803 7862 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 109bdf09ab804bf1b13ca35cce2e6bcf [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 05:17:54.887998 7871 sys_catalog.cc:455] T 00000000000000000000000000000000 P 599975ad9f374b9cba8b0d1e09b84822 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 599975ad9f374b9cba8b0d1e09b84822. Latest consensus state: current_term: 2 leader_uuid: "599975ad9f374b9cba8b0d1e09b84822" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "401619c49c134f59a36d3d960b3589ee" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38331 } } peers { permanent_uuid: "109bdf09ab804bf1b13ca35cce2e6bcf" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40533 } attrs { promote: false } } peers { permanent_uuid: "599975ad9f374b9cba8b0d1e09b84822" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39701 } attrs { promote: false } } }
I20260928 05:17:54.888383 7871 sys_catalog.cc:458] T 00000000000000000000000000000000 P 599975ad9f374b9cba8b0d1e09b84822 [sys.catalog]: This master's current role is: LEADER
I20260928 05:17:54.889034 7873 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:17:54.892225 7873 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:17:54.893013 7873 catalog_manager.cc:1259] Loaded cluster ID: 617aa3f5fc464ce4a48eb676675ac7d8
I20260928 05:17:54.893077 7873 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:17:54.894098 7873 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:17:54.894882 7873 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 599975ad9f374b9cba8b0d1e09b84822: Loaded TSK: 0
I20260928 05:17:54.895625 7873 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 05:17:54.944404 7325 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51892:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 05:17:54.951510 7325 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 05:17:54.955216 7864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:54.954851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38331 (local address 127.0.0.1:32844)
0928 05:17:54.955018 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:54.955084 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 05:17:54.955250 7146 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 109bdf09ab804bf1b13ca35cce2e6bcf [term 2 FOLLOWER]: Refusing update from remote peer 599975ad9f374b9cba8b0d1e09b84822: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260928 05:17:54.955956 7871 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 599975ad9f374b9cba8b0d1e09b84822 [LEADER]: Connected to new peer: Peer: permanent_uuid: "109bdf09ab804bf1b13ca35cce2e6bcf" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40533 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260928 05:17:54.956087 7864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:54.955729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38331 (local address 127.0.0.1:32852)
0928 05:17:54.955873 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:54.955945 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260928 05:17:54.956259 7309 consensus_peers.cc:487] T 00000000000000000000000000000000 P 599975ad9f374b9cba8b0d1e09b84822 -> Peer 401619c49c134f59a36d3d960b3589ee (127.0.97.62:38331): Couldn't send request to peer 401619c49c134f59a36d3d960b3589ee. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38331: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 05:17:54.958904 7862 sys_catalog.cc:455] T 00000000000000000000000000000000 P 109bdf09ab804bf1b13ca35cce2e6bcf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 599975ad9f374b9cba8b0d1e09b84822. Latest consensus state: current_term: 2 leader_uuid: "599975ad9f374b9cba8b0d1e09b84822" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "401619c49c134f59a36d3d960b3589ee" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38331 } } peers { permanent_uuid: "109bdf09ab804bf1b13ca35cce2e6bcf" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40533 } attrs { promote: false } } peers { permanent_uuid: "599975ad9f374b9cba8b0d1e09b84822" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39701 } attrs { promote: false } } }
I20260928 05:17:54.959165 7862 sys_catalog.cc:458] T 00000000000000000000000000000000 P 109bdf09ab804bf1b13ca35cce2e6bcf [sys.catalog]: This master's current role is: FOLLOWER
I20260928 05:17:54.959419 7874 mvcc.cc:205] Tried to move back new op lower bound from 7334185676611022848 to 7334185676336693248. Current Snapshot: MvccSnapshot[applied={T|T < 7334185676611022848}]
I20260928 05:17:54.960052 7871 sys_catalog.cc:455] T 00000000000000000000000000000000 P 599975ad9f374b9cba8b0d1e09b84822 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "599975ad9f374b9cba8b0d1e09b84822" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "401619c49c134f59a36d3d960b3589ee" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38331 } } peers { permanent_uuid: "109bdf09ab804bf1b13ca35cce2e6bcf" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40533 } attrs { promote: false } } peers { permanent_uuid: "599975ad9f374b9cba8b0d1e09b84822" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39701 } attrs { promote: false } } }
I20260928 05:17:54.960333 7871 sys_catalog.cc:458] T 00000000000000000000000000000000 P 599975ad9f374b9cba8b0d1e09b84822 [sys.catalog]: This master's current role is: LEADER
I20260928 05:17:54.961311 7862 sys_catalog.cc:455] T 00000000000000000000000000000000 P 109bdf09ab804bf1b13ca35cce2e6bcf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "599975ad9f374b9cba8b0d1e09b84822" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "401619c49c134f59a36d3d960b3589ee" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38331 } } peers { permanent_uuid: "109bdf09ab804bf1b13ca35cce2e6bcf" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40533 } attrs { promote: false } } peers { permanent_uuid: "599975ad9f374b9cba8b0d1e09b84822" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39701 } attrs { promote: false } } }
I20260928 05:17:54.961585 7862 sys_catalog.cc:458] T 00000000000000000000000000000000 P 109bdf09ab804bf1b13ca35cce2e6bcf [sys.catalog]: This master's current role is: FOLLOWER
I20260928 05:17:54.974074 7864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:54.967342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33089 (local address 127.0.0.1:47890)
0928 05:17:54.967790 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:54.967803 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:54.967873 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:54.968291 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:54.968297 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:54.968326 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:17:54.968545 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:54.968555 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:54.970223 (+ 1668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:54.970229 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:54.971775 (+ 1546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:54.971788 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:54.973363 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:54.973368 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:54.973583 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:54.973616 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 05:17:54.973681 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 05:17:54.973805 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365}
I20260928 05:17:54.974694 7863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:54.966998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43633 (local address 127.0.0.1:32934)
0928 05:17:54.967463 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:54.967481 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:54.967605 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:54.969107 (+ 1502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:54.969112 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:54.969136 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:17:54.969434 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:54.969446 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:54.971001 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:54.971007 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:54.972538 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:54.972548 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:54.973957 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:54.973963 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:54.974193 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:54.974225 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 05:17:54.974314 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 05:17:54.974456 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"mutex_wait_us":17}
I20260928 05:17:54.974740 7880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:54.967148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32934 (local address 127.0.97.1:43633)
0928 05:17:54.968757 (+ 1609us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:54.968762 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:54.968780 (+ 18us) server_negotiation.cc:407] Connection header received
0928 05:17:54.968843 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:54.968870 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:54.968942 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:54.969573 (+ 631us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:17:54.969743 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:54.970868 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:54.972675 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:54.973737 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:54.974212 (+ 475us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:54.974307 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:54.974353 (+ 46us) server_negotiation.cc:299] Negotiation successful
0928 05:17:54.974480 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1492,"thread_start_us":1395,"threads_started":1}
I20260928 05:17:54.974768 7881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:54.967462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47890 (local address 127.0.97.2:33089)
0928 05:17:54.967944 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:54.967952 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:54.967972 (+ 20us) server_negotiation.cc:407] Connection header received
0928 05:17:54.968046 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:54.968051 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:54.968127 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:54.969255 (+ 1128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:17:54.969419 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:54.970095 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:54.971950 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:54.973207 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:54.974374 (+ 1167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:54.974448 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:54.974495 (+ 47us) server_negotiation.cc:299] Negotiation successful
0928 05:17:54.974586 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":239,"threads_started":1}
I20260928 05:17:54.975399 7438 tablet_service.cc:1467] Processing CreateTablet for tablet 57665c354e9849f48253c60ca761bba4 (DEFAULT_TABLE table=test-table [id=0e6fbca7eb5444a0bbbbb47826a510c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:17:54.975399 7582 tablet_service.cc:1467] Processing CreateTablet for tablet 57665c354e9849f48253c60ca761bba4 (DEFAULT_TABLE table=test-table [id=0e6fbca7eb5444a0bbbbb47826a510c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:17:54.975790 7438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57665c354e9849f48253c60ca761bba4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:17:54.975801 7582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57665c354e9849f48253c60ca761bba4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:17:54.980630 7882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:54.968270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46223 (local address 127.0.0.1:40588)
0928 05:17:54.973250 (+ 4980us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:54.973267 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:54.973375 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:54.973910 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:54.973914 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:54.973940 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:17:54.975229 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:54.975242 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:54.976486 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:54.976491 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:54.978372 (+ 1881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:54.978385 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:54.979833 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:54.979839 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:54.980001 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:54.980034 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 05:17:54.980123 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 05:17:54.980240 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4868,"mutex_wait_us":4,"thread_start_us":4715,"threads_started":1}
I20260928 05:17:54.981571 7883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:54.968661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40588 (local address 127.0.97.3:46223)
0928 05:17:54.969100 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:54.969106 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:54.973502 (+ 4396us) server_negotiation.cc:407] Connection header received
0928 05:17:54.973576 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:54.973581 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:54.973661 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:54.973747 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:17:54.975390 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:54.976339 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:54.978515 (+ 2176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:54.979647 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:54.979761 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:54.979863 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:54.980168 (+ 305us) server_negotiation.cc:299] Negotiation successful
0928 05:17:54.981257 (+ 1089us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":396,"threads_started":1}
I20260928 05:17:54.981812 7884 tablet_bootstrap.cc:492] T 57665c354e9849f48253c60ca761bba4 P 176eae27f2854c7a84d8a0e5c342d0a9: Bootstrap starting.
I20260928 05:17:54.982318 7722 tablet_service.cc:1467] Processing CreateTablet for tablet 57665c354e9849f48253c60ca761bba4 (DEFAULT_TABLE table=test-table [id=0e6fbca7eb5444a0bbbbb47826a510c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:17:54.982687 7722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57665c354e9849f48253c60ca761bba4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:17:54.983348 7884 tablet_bootstrap.cc:654] T 57665c354e9849f48253c60ca761bba4 P 176eae27f2854c7a84d8a0e5c342d0a9: Neither blocks nor log segments found. Creating new log.
I20260928 05:17:54.985297 7884 tablet_bootstrap.cc:492] T 57665c354e9849f48253c60ca761bba4 P 176eae27f2854c7a84d8a0e5c342d0a9: No bootstrap required, opened a new log
I20260928 05:17:54.985433 7884 ts_tablet_manager.cc:1397] T 57665c354e9849f48253c60ca761bba4 P 176eae27f2854c7a84d8a0e5c342d0a9: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260928 05:17:54.986068 7886 tablet_bootstrap.cc:492] T 57665c354e9849f48253c60ca761bba4 P dafeb19659594f5abdc1fc79acdfb9df: Bootstrap starting.
I20260928 05:17:54.986142 7884 raft_consensus.cc:348] T 57665c354e9849f48253c60ca761bba4 P 176eae27f2854c7a84d8a0e5c342d0a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dafeb19659594f5abdc1fc79acdfb9df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43633 } } peers { permanent_uuid: "176eae27f2854c7a84d8a0e5c342d0a9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33089 } } peers { permanent_uuid: "a2d63277ac00458a83c7cb684be08738" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46223 } }
I20260928 05:17:54.986358 7884 raft_consensus.cc:374] T 57665c354e9849f48253c60ca761bba4 P 176eae27f2854c7a84d8a0e5c342d0a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:17:54.986485 7884 raft_consensus.cc:729] T 57665c354e9849f48253c60ca761bba4 P 176eae27f2854c7a84d8a0e5c342d0a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 176eae27f2854c7a84d8a0e5c342d0a9, State: Initialized, Role: FOLLOWER
I20260928 05:17:54.986698 7884 consensus_queue.cc:260] T 57665c354e9849f48253c60ca761bba4 P 176eae27f2854c7a84d8a0e5c342d0a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dafeb19659594f5abdc1fc79acdfb9df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43633 } } peers { permanent_uuid: "176eae27f2854c7a84d8a0e5c342d0a9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33089 } } peers { permanent_uuid: "a2d63277ac00458a83c7cb684be08738" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46223 } }
I20260928 05:17:54.987401 7884 ts_tablet_manager.cc:1428] T 57665c354e9849f48253c60ca761bba4 P 176eae27f2854c7a84d8a0e5c342d0a9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 05:17:54.987604 7886 tablet_bootstrap.cc:654] T 57665c354e9849f48253c60ca761bba4 P dafeb19659594f5abdc1fc79acdfb9df: Neither blocks nor log segments found. Creating new log.
I20260928 05:17:54.990401 7647 heartbeater.cc:503] Master 127.0.97.60:39701 was elected leader, sending a full tablet report...
I20260928 05:17:54.997704 7889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:54.987932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38331 (local address 127.0.97.2:46093)
0928 05:17:54.997317 (+ 9385us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:54.997440 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":9250,"thread_start_us":9105,"threads_started":1}
I20260928 05:17:54.998731 7889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:54.998410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38331 (local address 127.0.97.2:54055)
0928 05:17:54.998550 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:54.998618 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 05:17:54.998790 7888 tablet_bootstrap.cc:492] T 57665c354e9849f48253c60ca761bba4 P a2d63277ac00458a83c7cb684be08738: Bootstrap starting.
I20260928 05:17:54.999011 7886 tablet_bootstrap.cc:492] T 57665c354e9849f48253c60ca761bba4 P dafeb19659594f5abdc1fc79acdfb9df: No bootstrap required, opened a new log
I20260928 05:17:54.999145 7886 ts_tablet_manager.cc:1397] T 57665c354e9849f48253c60ca761bba4 P dafeb19659594f5abdc1fc79acdfb9df: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.010s
I20260928 05:17:54.999879 7886 raft_consensus.cc:348] T 57665c354e9849f48253c60ca761bba4 P dafeb19659594f5abdc1fc79acdfb9df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dafeb19659594f5abdc1fc79acdfb9df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43633 } } peers { permanent_uuid: "176eae27f2854c7a84d8a0e5c342d0a9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33089 } } peers { permanent_uuid: "a2d63277ac00458a83c7cb684be08738" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46223 } }
I20260928 05:17:55.000135 7886 raft_consensus.cc:374] T 57665c354e9849f48253c60ca761bba4 P dafeb19659594f5abdc1fc79acdfb9df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:17:55.000233 7886 raft_consensus.cc:729] T 57665c354e9849f48253c60ca761bba4 P dafeb19659594f5abdc1fc79acdfb9df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dafeb19659594f5abdc1fc79acdfb9df, State: Initialized, Role: FOLLOWER
I20260928 05:17:55.000404 7888 tablet_bootstrap.cc:654] T 57665c354e9849f48253c60ca761bba4 P a2d63277ac00458a83c7cb684be08738: Neither blocks nor log segments found. Creating new log.
I20260928 05:17:55.000487 7886 consensus_queue.cc:260] T 57665c354e9849f48253c60ca761bba4 P dafeb19659594f5abdc1fc79acdfb9df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dafeb19659594f5abdc1fc79acdfb9df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43633 } } peers { permanent_uuid: "176eae27f2854c7a84d8a0e5c342d0a9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33089 } } peers { permanent_uuid: "a2d63277ac00458a83c7cb684be08738" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46223 } }
I20260928 05:17:55.001288 7886 ts_tablet_manager.cc:1428] T 57665c354e9849f48253c60ca761bba4 P dafeb19659594f5abdc1fc79acdfb9df: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260928 05:17:55.003217 7503 heartbeater.cc:503] Master 127.0.97.60:39701 was elected leader, sending a full tablet report...
I20260928 05:17:55.009708 7893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:55.001691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38331 (local address 127.0.97.1:48675)
0928 05:17:55.009280 (+ 7589us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:55.009421 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":7464,"thread_start_us":7334,"threads_started":1}
I20260928 05:17:55.010562 7893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:55.010266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38331 (local address 127.0.97.1:38849)
0928 05:17:55.010393 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:55.010456 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 05:17:55.010644 7888 tablet_bootstrap.cc:492] T 57665c354e9849f48253c60ca761bba4 P a2d63277ac00458a83c7cb684be08738: No bootstrap required, opened a new log
I20260928 05:17:55.010782 7888 ts_tablet_manager.cc:1397] T 57665c354e9849f48253c60ca761bba4 P a2d63277ac00458a83c7cb684be08738: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.004s
I20260928 05:17:55.011518 7888 raft_consensus.cc:348] T 57665c354e9849f48253c60ca761bba4 P a2d63277ac00458a83c7cb684be08738 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dafeb19659594f5abdc1fc79acdfb9df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43633 } } peers { permanent_uuid: "176eae27f2854c7a84d8a0e5c342d0a9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33089 } } peers { permanent_uuid: "a2d63277ac00458a83c7cb684be08738" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46223 } }
I20260928 05:17:55.011693 7888 raft_consensus.cc:374] T 57665c354e9849f48253c60ca761bba4 P a2d63277ac00458a83c7cb684be08738 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:17:55.011770 7888 raft_consensus.cc:729] T 57665c354e9849f48253c60ca761bba4 P a2d63277ac00458a83c7cb684be08738 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2d63277ac00458a83c7cb684be08738, State: Initialized, Role: FOLLOWER
I20260928 05:17:55.011996 7888 consensus_queue.cc:260] T 57665c354e9849f48253c60ca761bba4 P a2d63277ac00458a83c7cb684be08738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dafeb19659594f5abdc1fc79acdfb9df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43633 } } peers { permanent_uuid: "176eae27f2854c7a84d8a0e5c342d0a9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33089 } } peers { permanent_uuid: "a2d63277ac00458a83c7cb684be08738" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46223 } }
I20260928 05:17:55.012580 7888 ts_tablet_manager.cc:1428] T 57665c354e9849f48253c60ca761bba4 P a2d63277ac00458a83c7cb684be08738: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260928 05:17:55.013711 7895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:55.012826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38331 (local address 127.0.97.3:33977)
0928 05:17:55.013353 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:55.013461 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":262,"threads_started":1}
I20260928 05:17:55.014148 7787 heartbeater.cc:503] Master 127.0.97.60:39701 was elected leader, sending a full tablet report...
I20260928 05:17:55.014808 7895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:55.014379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38331 (local address 127.0.97.3:47997)
0928 05:17:55.014621 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:55.014692 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260928 05:17:55.123473 7506 tablet.cc:2369] T 57665c354e9849f48253c60ca761bba4 P dafeb19659594f5abdc1fc79acdfb9df: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 05:17:55.234895 7887 raft_consensus.cc:482] T 57665c354e9849f48253c60ca761bba4 P 176eae27f2854c7a84d8a0e5c342d0a9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:17:55.235093 7887 raft_consensus.cc:504] T 57665c354e9849f48253c60ca761bba4 P 176eae27f2854c7a84d8a0e5c342d0a9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dafeb19659594f5abdc1fc79acdfb9df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43633 } } peers { permanent_uuid: "176eae27f2854c7a84d8a0e5c342d0a9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33089 } } peers { permanent_uuid: "a2d63277ac00458a83c7cb684be08738" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46223 } }
I20260928 05:17:55.235767 7887 leader_election.cc:288] T 57665c354e9849f48253c60ca761bba4 P 176eae27f2854c7a84d8a0e5c342d0a9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dafeb19659594f5abdc1fc79acdfb9df (127.0.97.1:43633), a2d63277ac00458a83c7cb684be08738 (127.0.97.3:46223)
I20260928 05:17:55.236276 7458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57665c354e9849f48253c60ca761bba4" candidate_uuid: "176eae27f2854c7a84d8a0e5c342d0a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dafeb19659594f5abdc1fc79acdfb9df" is_pre_election: true
I20260928 05:17:55.236274 7742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57665c354e9849f48253c60ca761bba4" candidate_uuid: "176eae27f2854c7a84d8a0e5c342d0a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2d63277ac00458a83c7cb684be08738" is_pre_election: true
I20260928 05:17:55.236498 7458 raft_consensus.cc:2454] T 57665c354e9849f48253c60ca761bba4 P dafeb19659594f5abdc1fc79acdfb9df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 176eae27f2854c7a84d8a0e5c342d0a9 in term 0.
I20260928 05:17:55.236510 7742 raft_consensus.cc:2454] T 57665c354e9849f48253c60ca761bba4 P a2d63277ac00458a83c7cb684be08738 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 176eae27f2854c7a84d8a0e5c342d0a9 in term 0.
I20260928 05:17:55.236987 7535 leader_election.cc:302] T 57665c354e9849f48253c60ca761bba4 P 176eae27f2854c7a84d8a0e5c342d0a9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 176eae27f2854c7a84d8a0e5c342d0a9, a2d63277ac00458a83c7cb684be08738, dafeb19659594f5abdc1fc79acdfb9df; no voters:
I20260928 05:17:55.237236 7887 raft_consensus.cc:2781] T 57665c354e9849f48253c60ca761bba4 P 176eae27f2854c7a84d8a0e5c342d0a9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 05:17:55.237329 7887 raft_consensus.cc:482] T 57665c354e9849f48253c60ca761bba4 P 176eae27f2854c7a84d8a0e5c342d0a9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 05:17:55.237392 7887 raft_consensus.cc:3037] T 57665c354e9849f48253c60ca761bba4 P 176eae27f2854c7a84d8a0e5c342d0a9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:17:55.239575 7887 raft_consensus.cc:504] T 57665c354e9849f48253c60ca761bba4 P 176eae27f2854c7a84d8a0e5c342d0a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dafeb19659594f5abdc1fc79acdfb9df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43633 } } peers { permanent_uuid: "176eae27f2854c7a84d8a0e5c342d0a9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33089 } } peers { permanent_uuid: "a2d63277ac00458a83c7cb684be08738" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46223 } }
I20260928 05:17:55.240095 7887 leader_election.cc:288] T 57665c354e9849f48253c60ca761bba4 P 176eae27f2854c7a84d8a0e5c342d0a9 [CANDIDATE]: Term 1 election: Requested vote from peers dafeb19659594f5abdc1fc79acdfb9df (127.0.97.1:43633), a2d63277ac00458a83c7cb684be08738 (127.0.97.3:46223)
I20260928 05:17:55.240515 7458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57665c354e9849f48253c60ca761bba4" candidate_uuid: "176eae27f2854c7a84d8a0e5c342d0a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dafeb19659594f5abdc1fc79acdfb9df"
I20260928 05:17:55.240573 7742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57665c354e9849f48253c60ca761bba4" candidate_uuid: "176eae27f2854c7a84d8a0e5c342d0a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2d63277ac00458a83c7cb684be08738"
I20260928 05:17:55.240700 7458 raft_consensus.cc:3037] T 57665c354e9849f48253c60ca761bba4 P dafeb19659594f5abdc1fc79acdfb9df [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:17:55.240722 7742 raft_consensus.cc:3037] T 57665c354e9849f48253c60ca761bba4 P a2d63277ac00458a83c7cb684be08738 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:17:55.243443 7742 raft_consensus.cc:2454] T 57665c354e9849f48253c60ca761bba4 P a2d63277ac00458a83c7cb684be08738 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 176eae27f2854c7a84d8a0e5c342d0a9 in term 1.
I20260928 05:17:55.243435 7458 raft_consensus.cc:2454] T 57665c354e9849f48253c60ca761bba4 P dafeb19659594f5abdc1fc79acdfb9df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 176eae27f2854c7a84d8a0e5c342d0a9 in term 1.
I20260928 05:17:55.243762 7535 leader_election.cc:302] T 57665c354e9849f48253c60ca761bba4 P 176eae27f2854c7a84d8a0e5c342d0a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 176eae27f2854c7a84d8a0e5c342d0a9, a2d63277ac00458a83c7cb684be08738; no voters:
I20260928 05:17:55.244017 7887 raft_consensus.cc:2781] T 57665c354e9849f48253c60ca761bba4 P 176eae27f2854c7a84d8a0e5c342d0a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:17:55.244372 7887 raft_consensus.cc:686] T 57665c354e9849f48253c60ca761bba4 P 176eae27f2854c7a84d8a0e5c342d0a9 [term 1 LEADER]: Becoming Leader. State: Replica: 176eae27f2854c7a84d8a0e5c342d0a9, State: Running, Role: LEADER
I20260928 05:17:55.244800 7887 consensus_queue.cc:237] T 57665c354e9849f48253c60ca761bba4 P 176eae27f2854c7a84d8a0e5c342d0a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dafeb19659594f5abdc1fc79acdfb9df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43633 } } peers { permanent_uuid: "176eae27f2854c7a84d8a0e5c342d0a9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33089 } } peers { permanent_uuid: "a2d63277ac00458a83c7cb684be08738" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46223 } }
I20260928 05:17:55.245390 7889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:55.244945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38331 (local address 127.0.97.2:36791)
0928 05:17:55.245111 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:55.245225 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260928 05:17:55.247210 7325 catalog_manager.cc:5495] T 57665c354e9849f48253c60ca761bba4 P 176eae27f2854c7a84d8a0e5c342d0a9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 176eae27f2854c7a84d8a0e5c342d0a9 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "176eae27f2854c7a84d8a0e5c342d0a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dafeb19659594f5abdc1fc79acdfb9df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43633 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "176eae27f2854c7a84d8a0e5c342d0a9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2d63277ac00458a83c7cb684be08738" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46223 } health_report { overall_health: UNKNOWN } } }
I20260928 05:17:55.271490 7881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:55.267685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47894 (local address 127.0.97.2:33089)
0928 05:17:55.267853 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:55.267858 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:55.267873 (+ 15us) server_negotiation.cc:407] Connection header received
0928 05:17:55.267999 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:55.268002 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:55.268069 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:55.268142 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:17:55.268408 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:55.269104 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:55.270286 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:55.270646 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:55.270730 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:55.271033 (+ 303us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:17:55.271118 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:17:55.271177 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:55.271266 (+ 89us) server_negotiation.cc:299] Negotiation successful
0928 05:17:55.271318 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 05:17:55.274045 7742 raft_consensus.cc:1261] T 57665c354e9849f48253c60ca761bba4 P a2d63277ac00458a83c7cb684be08738 [term 1 FOLLOWER]: Refusing update from remote peer 176eae27f2854c7a84d8a0e5c342d0a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:17:55.274076 7458 raft_consensus.cc:1261] T 57665c354e9849f48253c60ca761bba4 P dafeb19659594f5abdc1fc79acdfb9df [term 1 FOLLOWER]: Refusing update from remote peer 176eae27f2854c7a84d8a0e5c342d0a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:17:55.274621 7896 consensus_queue.cc:1035] T 57665c354e9849f48253c60ca761bba4 P 176eae27f2854c7a84d8a0e5c342d0a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2d63277ac00458a83c7cb684be08738" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:17:55.274885 7887 consensus_queue.cc:1035] T 57665c354e9849f48253c60ca761bba4 P 176eae27f2854c7a84d8a0e5c342d0a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dafeb19659594f5abdc1fc79acdfb9df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:17:55.276778 7895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:55.276319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38331 (local address 127.0.97.3:51491)
0928 05:17:55.276525 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:55.276619 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260928 05:17:55.277176 7893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:55.276662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38331 (local address 127.0.97.1:44167)
0928 05:17:55.276886 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:55.276998 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260928 05:17:55.284196 7889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:55.283625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38331 (local address 127.0.97.2:60699)
0928 05:17:55.283994 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:55.284075 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":263}
I20260928 05:17:55.286127 7893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:55.283849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38331 (local address 127.0.97.1:44775)
0928 05:17:55.285950 (+ 2101us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:55.286021 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2008}
I20260928 05:17:55.287950 7901 mvcc.cc:205] Tried to move back new op lower bound from 7334185677917270016 to 7334185677805187072. Current Snapshot: MvccSnapshot[applied={T|T < 7334185677917270016}]
I20260928 05:17:55.288687 7889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:55.288260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38331 (local address 127.0.97.2:38967)
0928 05:17:55.288431 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:55.288524 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260928 05:17:55.289484 7895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:55.289034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38331 (local address 127.0.97.3:59725)
0928 05:17:55.289223 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:55.289309 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260928 05:17:55.290527 7325 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51892:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:17:55.290750 7325 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51892:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:17:55.296129 7325 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 599975ad9f374b9cba8b0d1e09b84822: Sending DeleteTablet for 3 replicas of tablet 57665c354e9849f48253c60ca761bba4
I20260928 05:17:55.296813 7438 tablet_service.cc:1514] Processing DeleteTablet for tablet 57665c354e9849f48253c60ca761bba4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:17:55 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:32934
I20260928 05:17:55.297444 7582 tablet_service.cc:1514] Processing DeleteTablet for tablet 57665c354e9849f48253c60ca761bba4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:17:55 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:47890
I20260928 05:17:55.297672 7722 tablet_service.cc:1514] Processing DeleteTablet for tablet 57665c354e9849f48253c60ca761bba4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:17:55 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:40588
I20260928 05:17:55.297629 7581 tablet_service.cc:1514] Processing DeleteTablet for tablet 57665c354e9849f48253c60ca761bba4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:17:55 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:47890
I20260928 05:17:55.297968 7721 tablet_service.cc:1514] Processing DeleteTablet for tablet 57665c354e9849f48253c60ca761bba4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:17:55 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:40588
I20260928 05:17:55.298761 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 7297
I20260928 05:17:55.300926 7908 tablet_replica.cc:330] T 57665c354e9849f48253c60ca761bba4 P dafeb19659594f5abdc1fc79acdfb9df: stopping tablet replica
I20260928 05:17:55.301035 7909 tablet_replica.cc:330] T 57665c354e9849f48253c60ca761bba4 P 176eae27f2854c7a84d8a0e5c342d0a9: stopping tablet replica
I20260928 05:17:55.301210 7908 raft_consensus.cc:2229] T 57665c354e9849f48253c60ca761bba4 P dafeb19659594f5abdc1fc79acdfb9df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:17:55.301258 7909 raft_consensus.cc:2229] T 57665c354e9849f48253c60ca761bba4 P 176eae27f2854c7a84d8a0e5c342d0a9 [term 1 LEADER]: Raft consensus shutting down.
I20260928 05:17:55.301524 7908 raft_consensus.cc:2258] T 57665c354e9849f48253c60ca761bba4 P dafeb19659594f5abdc1fc79acdfb9df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:17:55.301672 7909 raft_consensus.cc:2258] T 57665c354e9849f48253c60ca761bba4 P 176eae27f2854c7a84d8a0e5c342d0a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:17:55.302695 7910 tablet_replica.cc:330] T 57665c354e9849f48253c60ca761bba4 P a2d63277ac00458a83c7cb684be08738: stopping tablet replica
I20260928 05:17:55.302745 7909 ts_tablet_manager.cc:1905] T 57665c354e9849f48253c60ca761bba4 P 176eae27f2854c7a84d8a0e5c342d0a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:17:55.302913 7910 raft_consensus.cc:2229] T 57665c354e9849f48253c60ca761bba4 P a2d63277ac00458a83c7cb684be08738 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:17:55.303246 7910 raft_consensus.cc:2258] T 57665c354e9849f48253c60ca761bba4 P a2d63277ac00458a83c7cb684be08738 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:17:55.304173 7910 ts_tablet_manager.cc:1905] T 57665c354e9849f48253c60ca761bba4 P a2d63277ac00458a83c7cb684be08738: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:17:55.304498 7908 ts_tablet_manager.cc:1905] T 57665c354e9849f48253c60ca761bba4 P dafeb19659594f5abdc1fc79acdfb9df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:17:55.309818 7909 ts_tablet_manager.cc:1918] T 57665c354e9849f48253c60ca761bba4 P 176eae27f2854c7a84d8a0e5c342d0a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:17:55.310012 7909 log.cc:1196] T 57665c354e9849f48253c60ca761bba4 P 176eae27f2854c7a84d8a0e5c342d0a9: Deleting WAL directory at /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/ts-1/wal/wals/57665c354e9849f48253c60ca761bba4
I20260928 05:17:55.310149 7910 ts_tablet_manager.cc:1918] T 57665c354e9849f48253c60ca761bba4 P a2d63277ac00458a83c7cb684be08738: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:17:55.310339 7910 log.cc:1196] T 57665c354e9849f48253c60ca761bba4 P a2d63277ac00458a83c7cb684be08738: Deleting WAL directory at /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/ts-2/wal/wals/57665c354e9849f48253c60ca761bba4
I20260928 05:17:55.310331 7908 ts_tablet_manager.cc:1918] T 57665c354e9849f48253c60ca761bba4 P dafeb19659594f5abdc1fc79acdfb9df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:17:55.310442 7908 log.cc:1196] T 57665c354e9849f48253c60ca761bba4 P dafeb19659594f5abdc1fc79acdfb9df: Deleting WAL directory at /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572622576160-388-0/minicluster-data/ts-0/wal/wals/57665c354e9849f48253c60ca761bba4
I20260928 05:17:55.310472 7909 ts_tablet_manager.cc:1939] T 57665c354e9849f48253c60ca761bba4 P 176eae27f2854c7a84d8a0e5c342d0a9: Deleting consensus metadata
I20260928 05:17:55.310880 7910 ts_tablet_manager.cc:1939] T 57665c354e9849f48253c60ca761bba4 P a2d63277ac00458a83c7cb684be08738: Deleting consensus metadata
I20260928 05:17:55.310894 7908 ts_tablet_manager.cc:1939] T 57665c354e9849f48253c60ca761bba4 P dafeb19659594f5abdc1fc79acdfb9df: Deleting consensus metadata
I20260928 05:17:55.311558 7309 catalog_manager.cc:4841] TS 176eae27f2854c7a84d8a0e5c342d0a9 (127.0.97.2:33089): tablet 57665c354e9849f48253c60ca761bba4 (table test-table [id=0e6fbca7eb5444a0bbbbb47826a510c9]) successfully deleted
I20260928 05:17:55.311853 7312 catalog_manager.cc:4841] TS dafeb19659594f5abdc1fc79acdfb9df (127.0.97.1:43633): tablet 57665c354e9849f48253c60ca761bba4 (table test-table [id=0e6fbca7eb5444a0bbbbb47826a510c9]) successfully deleted
W20260928 05:17:55.312353 7309 catalog_manager.cc:4808] TS 176eae27f2854c7a84d8a0e5c342d0a9 (127.0.97.2:33089): delete failed for tablet 57665c354e9849f48253c60ca761bba4 because the tablet was not found. No further retry: Not found: Tablet not found: 57665c354e9849f48253c60ca761bba4
I20260928 05:17:55.313149 7309 catalog_manager.cc:4841] TS a2d63277ac00458a83c7cb684be08738 (127.0.97.3:46223): tablet 57665c354e9849f48253c60ca761bba4 (table test-table [id=0e6fbca7eb5444a0bbbbb47826a510c9]) successfully deleted
W20260928 05:17:55.313406 7309 catalog_manager.cc:4808] TS a2d63277ac00458a83c7cb684be08738 (127.0.97.3:46223): delete failed for tablet 57665c354e9849f48253c60ca761bba4 because the tablet was not found. No further retry: Not found: Tablet not found: 57665c354e9849f48253c60ca761bba4
I20260928 05:17:55.316314 7880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:55.299570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32946 (local address 127.0.97.1:43633)
0928 05:17:55.299747 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:55.299754 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:55.302228 (+ 2474us) server_negotiation.cc:407] Connection header received
0928 05:17:55.302372 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:55.302377 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:55.302448 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:55.302535 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:55.304228 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:55.305340 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:55.307033 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:55.307666 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:55.307761 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:55.311123 (+ 3362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:55.311141 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:55.311160 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:55.311206 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:55.314228 (+ 3022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:55.314805 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:55.314812 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:55.314814 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:55.314920 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:55.315202 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:55.315205 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:55.315207 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:55.315500 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:55.315696 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:55.315869 (+ 173us) server_negotiation.cc:299] Negotiation successful
0928 05:17:55.316032 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 05:17:55.316900 7498 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:17:55.405229 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 7297
I20260928 05:17:55.418570 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 7514
I20260928 05:17:55.430466 7881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:55.419178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47898 (local address 127.0.97.2:33089)
0928 05:17:55.419353 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:55.419359 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:55.419435 (+ 76us) server_negotiation.cc:407] Connection header received
0928 05:17:55.419611 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:55.419615 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:55.419676 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:55.419757 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:55.420944 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:55.421645 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:55.422831 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:55.423212 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:55.423315 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:55.425648 (+ 2333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:55.425661 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:55.425677 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:55.425709 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:55.428049 (+ 2340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:55.428630 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:55.428637 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:55.428639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:55.428761 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:55.429118 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:55.429121 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:55.429123 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:55.429429 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:55.429713 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:55.429986 (+ 273us) server_negotiation.cc:299] Negotiation successful
0928 05:17:55.430180 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 05:17:55.431120 7642 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:17:55.544200 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 7514
I20260928 05:17:55.549762 7914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:55.549079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38331 (local address 127.0.0.1:32854)
0928 05:17:55.549517 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:55.549607 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":238,"threads_started":1}
I20260928 05:17:55.550889 7914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:55.550592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38331 (local address 127.0.0.1:32868)
0928 05:17:55.550726 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:55.550788 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260928 05:17:55.561987 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 7659
I20260928 05:17:55.573550 7916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:55.562650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40600 (local address 127.0.97.3:46223)
0928 05:17:55.563145 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:55.563149 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:55.563163 (+ 14us) server_negotiation.cc:407] Connection header received
0928 05:17:55.563212 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:55.563216 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:55.563268 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:55.563348 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:55.564605 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:55.565328 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:55.566429 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:55.566788 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:55.566869 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:55.569022 (+ 2153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:55.569035 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:55.569050 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:55.569077 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:55.571439 (+ 2362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:55.571984 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:55.571996 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:55.571998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:55.572113 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:55.572388 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:55.572391 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:55.572393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:55.572700 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:55.572961 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:55.573160 (+ 199us) server_negotiation.cc:299] Negotiation successful
0928 05:17:55.573311 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":308,"threads_started":1}
I20260928 05:17:55.574131 7782 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:17:55.665834 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 7659
I20260928 05:17:55.678925 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 7000
I20260928 05:17:55.679932 7166 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:17:55.790545 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 7000
W20260928 05:17:55.806356 7310 connection.cc:535] client connection to 127.0.97.61:40533 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 05:17:55.806648 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 7100
I20260928 05:17:55.807387 7914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:55.806971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40533 (local address 127.0.0.1:52300)
0928 05:17:55.807148 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:55.807231 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260928 05:17:55.807546 7355 generic_service.cc:196] Checking for leaks (request via RPC)
W20260928 05:17:55.807587 7310 consensus_peers.cc:487] T 00000000000000000000000000000000 P 599975ad9f374b9cba8b0d1e09b84822 -> Peer 109bdf09ab804bf1b13ca35cce2e6bcf (127.0.97.61:40533): Couldn't send request to peer 109bdf09ab804bf1b13ca35cce2e6bcf. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40533: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 05:17:55.921264 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 7100
2026-09-28T05:17:55Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (8900 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:17:56 dist-test-slave-whsn krb5kdc[7922](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:17:56 dist-test-slave-whsn krb5kdc[7922](info): set up 2 sockets
Sep 28 05:17:56 dist-test-slave-whsn krb5kdc[7922](info): commencing operation
krb5kdc: starting...
W20260928 05:17:56.925298 388 mini_kdc.cc:121] Time spent starting KDC: real 0.965s user 0.004s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/krb5kdc/test-user.keytab.
Sep 28 05:17:57 dist-test-slave-whsn krb5kdc[7922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572677, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:17:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:17:57Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260928 05:17:57.103895 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39261
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39261
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:17:57.399217 7935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:17:57.399473 7935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:17:57.399593 7935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:17:57.406461 7935 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:17:57.406550 7935 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 05:17:57.406612 7935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:17:57.406694 7935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:17:57.406754 7935 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:17:57.406832 7935 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:17:57.415992 7935 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39261
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:17:57.417132 7935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:17:57.418391 7935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:17:57.428750 7940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:57.428701 7941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:57.428717 7943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:17:57.429430 7935 server_base.cc:1034] running on GCE node
I20260928 05:17:57.430066 7935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:17:57.431206 7935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:17:57.432402 7935 hybrid_clock.cc:648] HybridClock initialized: now 1790572677432339 us; error 36 us; skew 500 ppm
I20260928 05:17:57.432689 7935 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:17:57 dist-test-slave-whsn krb5kdc[7922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572677, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:17:57.435380 7935 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260928 05:17:57.436764 7935 webserver.cc:460] Webserver started at http://127.0.97.62:44987/ using document root <none> and password file <none>
I20260928 05:17:57.437394 7935 fs_manager.cc:359] Metadata directory not provided
I20260928 05:17:57.437484 7935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:17:57.437767 7935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:17:57.440344 7935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/minicluster-data/master-0/data/instance:
uuid: "ca2877e519c94fd181176dfe63838bb3"
format_stamp: "Formatted at 2026-09-28 05:17:57 on dist-test-slave-whsn"
server_key: "2a9938391e0b022b2ace7b618f695d8b"
server_key_iv: "e6f98d4e66ebd1f307ce86579a098589"
server_key_version: "encryptionkey@0"
I20260928 05:17:57.441097 7935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/minicluster-data/master-0/wal/instance:
uuid: "ca2877e519c94fd181176dfe63838bb3"
format_stamp: "Formatted at 2026-09-28 05:17:57 on dist-test-slave-whsn"
server_key: "2a9938391e0b022b2ace7b618f695d8b"
server_key_iv: "e6f98d4e66ebd1f307ce86579a098589"
server_key_version: "encryptionkey@0"
I20260928 05:17:57.445604 7935 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 05:17:57.448558 7950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:57.449728 7935 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 05:17:57.449867 7935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/minicluster-data/master-0/wal
uuid: "ca2877e519c94fd181176dfe63838bb3"
format_stamp: "Formatted at 2026-09-28 05:17:57 on dist-test-slave-whsn"
server_key: "2a9938391e0b022b2ace7b618f695d8b"
server_key_iv: "e6f98d4e66ebd1f307ce86579a098589"
server_key_version: "encryptionkey@0"
I20260928 05:17:57.450039 7935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:17:57.472641 7935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:17:57.476092 7935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:17:57.476300 7935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:17:57.494630 7935 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39261
I20260928 05:17:57.494658 8011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39261 every 8 connection(s)
I20260928 05:17:57.495770 7935 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:17:57.499193 8012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:17:57.499190 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 7935
I20260928 05:17:57.499374 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/minicluster-data/master-0/wal/instance
I20260928 05:17:57.499948 388 external_mini_cluster.cc:1428] Setting key 00b312133421280100e4514ba54377a1
I20260928 05:17:57.511374 8012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca2877e519c94fd181176dfe63838bb3: Bootstrap starting.
Sep 28 05:17:57 dist-test-slave-whsn krb5kdc[7922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572677, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260928 05:17:57.514816 8012 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca2877e519c94fd181176dfe63838bb3: Neither blocks nor log segments found. Creating new log.
I20260928 05:17:57.515761 8012 log.cc:824] T 00000000000000000000000000000000 P ca2877e519c94fd181176dfe63838bb3: Log is configured to *not* fsync() on all Append() calls
I20260928 05:17:57.518499 8012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca2877e519c94fd181176dfe63838bb3: No bootstrap required, opened a new log
I20260928 05:17:57.521878 8015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:57.501661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53234 (local address 127.0.97.62:39261)
0928 05:17:57.502334 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:57.502345 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:57.502382 (+ 37us) server_negotiation.cc:407] Connection header received
0928 05:17:57.504506 (+ 2124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:57.504526 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:57.504938 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:57.505241 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:57.506756 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:57.507956 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:57.509650 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:57.510248 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:57.510361 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:57.514654 (+ 4293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:57.514662 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:57.514675 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:57.514708 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:57.517840 (+ 3132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:57.518424 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:57.518432 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:57.518439 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:57.518705 (+ 266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:57.519022 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:57.519027 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:57.519030 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:57.519633 (+ 603us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:57.519921 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:57.520204 (+ 283us) server_negotiation.cc:299] Negotiation successful
0928 05:17:57.520598 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":327,"threads_started":1}
I20260928 05:17:57.526904 8012 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca2877e519c94fd181176dfe63838bb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca2877e519c94fd181176dfe63838bb3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39261 } }
I20260928 05:17:57.527189 8012 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca2877e519c94fd181176dfe63838bb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:17:57.527254 8012 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca2877e519c94fd181176dfe63838bb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca2877e519c94fd181176dfe63838bb3, State: Initialized, Role: FOLLOWER
I20260928 05:17:57.527688 8012 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca2877e519c94fd181176dfe63838bb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca2877e519c94fd181176dfe63838bb3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39261 } }
I20260928 05:17:57.527872 8012 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca2877e519c94fd181176dfe63838bb3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:17:57.527961 8012 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca2877e519c94fd181176dfe63838bb3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:17:57.528075 8012 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca2877e519c94fd181176dfe63838bb3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:17:57.530032 8012 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca2877e519c94fd181176dfe63838bb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca2877e519c94fd181176dfe63838bb3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39261 } }
I20260928 05:17:57.530529 8012 leader_election.cc:302] T 00000000000000000000000000000000 P ca2877e519c94fd181176dfe63838bb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca2877e519c94fd181176dfe63838bb3; no voters:
I20260928 05:17:57.531015 8012 leader_election.cc:288] T 00000000000000000000000000000000 P ca2877e519c94fd181176dfe63838bb3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:17:57.531124 8017 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca2877e519c94fd181176dfe63838bb3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:17:57.531904 8017 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca2877e519c94fd181176dfe63838bb3 [term 1 LEADER]: Becoming Leader. State: Replica: ca2877e519c94fd181176dfe63838bb3, State: Running, Role: LEADER
I20260928 05:17:57.532352 8017 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca2877e519c94fd181176dfe63838bb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca2877e519c94fd181176dfe63838bb3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39261 } }
I20260928 05:17:57.532785 8012 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca2877e519c94fd181176dfe63838bb3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:17:57.534622 8018 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca2877e519c94fd181176dfe63838bb3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca2877e519c94fd181176dfe63838bb3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca2877e519c94fd181176dfe63838bb3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39261 } } }
I20260928 05:17:57.534976 8018 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca2877e519c94fd181176dfe63838bb3 [sys.catalog]: This master's current role is: LEADER
I20260928 05:17:57.536643 8019 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca2877e519c94fd181176dfe63838bb3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca2877e519c94fd181176dfe63838bb3. Latest consensus state: current_term: 1 leader_uuid: "ca2877e519c94fd181176dfe63838bb3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca2877e519c94fd181176dfe63838bb3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39261 } } }
I20260928 05:17:57.536906 8023 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:17:57.536952 8019 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca2877e519c94fd181176dfe63838bb3 [sys.catalog]: This master's current role is: LEADER
I20260928 05:17:57.541426 8023 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:17:57.549691 8023 catalog_manager.cc:1347] Generated new cluster ID: 8eeaa4d0db374bd3ad54413fb89c6a3d
I20260928 05:17:57.549783 8023 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:17:57.589860 8023 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:17:57.591126 8023 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:17:57.602285 8023 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca2877e519c94fd181176dfe63838bb3: Generated new TSK 0
I20260928 05:17:57.603055 8023 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260928 05:17:57.656303 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39261
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:17:57.956610 8038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:17:57.956871 8038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:17:57.957010 8038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:17:57.963770 8038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:17:57.963864 8038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:17:57.963985 8038 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 05:17:57.964083 8038 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 05:17:57.973228 8038 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:39261
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:17:57.974421 8038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:17:57.975649 8038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:17:57.987313 8043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:57.987279 8046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:57.987357 8044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:17:57.988205 8038 server_base.cc:1034] running on GCE node
I20260928 05:17:57.988727 8038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:17:57.989544 8038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:17:57.990746 8038 hybrid_clock.cc:648] HybridClock initialized: now 1790572677990693 us; error 30 us; skew 500 ppm
I20260928 05:17:57.991005 8038 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:17:57 dist-test-slave-whsn krb5kdc[7922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572677, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:17:57.993816 8038 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260928 05:17:57.995031 8038 webserver.cc:460] Webserver started at http://127.0.97.1:35211/ using document root <none> and password file <none>
I20260928 05:17:57.995625 8038 fs_manager.cc:359] Metadata directory not provided
I20260928 05:17:57.995714 8038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:17:57.995988 8038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:17:57.998617 8038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/minicluster-data/ts-0/data/instance:
uuid: "0f89f414d76944cfb5a9505edaa66d35"
format_stamp: "Formatted at 2026-09-28 05:17:57 on dist-test-slave-whsn"
server_key: "8f543805aa25a9a92f1ba0b37d285c0b"
server_key_iv: "b775c0d2b9a44df9db5e2b37b01850cc"
server_key_version: "encryptionkey@0"
I20260928 05:17:57.999306 8038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/minicluster-data/ts-0/wal/instance:
uuid: "0f89f414d76944cfb5a9505edaa66d35"
format_stamp: "Formatted at 2026-09-28 05:17:57 on dist-test-slave-whsn"
server_key: "8f543805aa25a9a92f1ba0b37d285c0b"
server_key_iv: "b775c0d2b9a44df9db5e2b37b01850cc"
server_key_version: "encryptionkey@0"
I20260928 05:17:58.003767 8038 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260928 05:17:58.006912 8053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:58.008075 8038 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260928 05:17:58.008221 8038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/minicluster-data/ts-0/wal
uuid: "0f89f414d76944cfb5a9505edaa66d35"
format_stamp: "Formatted at 2026-09-28 05:17:57 on dist-test-slave-whsn"
server_key: "8f543805aa25a9a92f1ba0b37d285c0b"
server_key_iv: "b775c0d2b9a44df9db5e2b37b01850cc"
server_key_version: "encryptionkey@0"
I20260928 05:17:58.008402 8038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:17:58.027258 8038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:17:58.030906 8038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:17:58.031127 8038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:17:58.033597 8038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:17:58.033675 8038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:58.033799 8038 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:17:58.033857 8038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 05:17:58 dist-test-slave-whsn krb5kdc[7922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572677, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260928 05:17:58.054768 8061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:58.033899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53609 (local address 127.0.97.62:39261)
0928 05:17:58.034675 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:58.034681 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:58.035533 (+ 852us) server_negotiation.cc:407] Connection header received
0928 05:17:58.037805 (+ 2272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:58.037811 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:58.037892 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:58.038023 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:58.040370 (+ 2347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:58.041095 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:58.043268 (+ 2173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:58.043881 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:58.044032 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:58.047913 (+ 3881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:58.047919 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:58.047924 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:58.047971 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:58.051196 (+ 3225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:58.052077 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:58.052080 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:58.052083 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:58.052199 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:58.052774 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:58.052779 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:58.052782 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:58.053159 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:58.053311 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:58.054258 (+ 947us) server_negotiation.cc:299] Negotiation successful
0928 05:17:58.054477 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":641,"thread_start_us":547,"threads_started":1}
I20260928 05:17:58.056510 8063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:58.034554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39261 (local address 127.0.97.1:53609)
0928 05:17:58.035358 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:58.035402 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:58.037516 (+ 2114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:58.038461 (+ 945us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:58.038472 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:58.039273 (+ 801us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:17:58.040178 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:58.040205 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:58.041242 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:58.041246 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:58.043125 (+ 1879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:58.043137 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:58.044140 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:58.044147 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:58.044337 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:58.044945 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:17:58.044970 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:17:58.047632 (+ 2662us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:17:58.051339 (+ 3707us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:58.051348 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:58.051366 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:58.051896 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:58.052302 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:58.052307 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:58.052311 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:58.052609 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:58.053296 (+ 687us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:17:58.053308 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:17:58.053738 (+ 430us) client_negotiation.cc:769] Sending connection context
0928 05:17:58.054086 (+ 348us) client_negotiation.cc:240] Negotiation successful
0928 05:17:58.055361 (+ 1275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":322,"threads_started":1}
I20260928 05:17:58.068948 8038 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42423
I20260928 05:17:58.069018 8173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42423 every 8 connection(s)
I20260928 05:17:58.070266 8038 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/minicluster-data/ts-0/data/info.pb
I20260928 05:17:58.072180 8174 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39261
I20260928 05:17:58.072362 8174 heartbeater.cc:464] Registering TS with master...
I20260928 05:17:58.072969 8174 heartbeater.cc:511] Master 127.0.97.62:39261 requested a full tablet report, sending...
I20260928 05:17:58.075083 7967 ts_manager.cc:194] Registered new tserver with Master: 0f89f414d76944cfb5a9505edaa66d35 (127.0.97.1:42423)
I20260928 05:17:58.075628 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 8038
I20260928 05:17:58.075765 8061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:58.062986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58931 (local address 127.0.97.62:39261)
0928 05:17:58.063150 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:58.063156 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:58.064082 (+ 926us) server_negotiation.cc:407] Connection header received
0928 05:17:58.064292 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:58.064298 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:58.064361 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:58.064409 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:58.065708 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:58.066567 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:58.068119 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:58.068577 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:58.068686 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:58.070510 (+ 1824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:58.070516 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:58.070520 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:58.070556 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:58.073297 (+ 2741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:58.073994 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:58.073999 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:58.074002 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:58.074178 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:58.074473 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:58.074478 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:58.074481 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:58.074800 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:58.074934 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:58.075267 (+ 333us) server_negotiation.cc:299] Negotiation successful
0928 05:17:58.075462 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 05:17:58.075781 8156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:58.063086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39261 (local address 127.0.97.1:58931)
0928 05:17:58.063966 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:58.063987 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:58.064131 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:58.064597 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:58.064601 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:58.065191 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:17:58.065581 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:58.065591 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:58.066752 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:58.066757 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:58.067955 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:58.067967 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:58.068806 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:58.068817 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:58.069006 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:58.069590 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:17:58.069605 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:17:58.070299 (+ 694us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:17:58.073409 (+ 3110us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:58.073414 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:58.073418 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:58.073866 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:58.074274 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:58.074277 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:58.074279 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:58.074382 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:58.074892 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:17:58.074894 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:17:58.075037 (+ 143us) client_negotiation.cc:769] Sending connection context
0928 05:17:58.075136 (+ 99us) client_negotiation.cc:240] Negotiation successful
0928 05:17:58.075382 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":707,"thread_start_us":567,"threads_started":1}
I20260928 05:17:58.075812 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/minicluster-data/ts-0/wal/instance
I20260928 05:17:58.076519 388 external_mini_cluster.cc:1428] Setting key a57e122f800f838305318a9957027621
I20260928 05:17:58.076961 7967 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:53609
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260928 05:17:58.115754 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39261
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:17:58.412117 8178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:17:58.412365 8178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:17:58.412477 8178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:17:58.420171 8178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:17:58.420275 8178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:17:58.420368 8178 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 05:17:58.420454 8178 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 05:17:58.430127 8178 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:39261
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:17:58.431275 8178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:17:58.432597 8178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:17:58.445150 8184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:58.445199 8183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:58.445278 8186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:17:58.446172 8178 server_base.cc:1034] running on GCE node
I20260928 05:17:58.446753 8178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:17:58.447811 8178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:17:58.449120 8178 hybrid_clock.cc:648] HybridClock initialized: now 1790572678449057 us; error 32 us; skew 500 ppm
I20260928 05:17:58.449373 8178 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:17:58 dist-test-slave-whsn krb5kdc[7922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572678, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:17:58.452186 8178 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260928 05:17:58.453516 8178 webserver.cc:460] Webserver started at http://127.0.97.2:44255/ using document root <none> and password file <none>
I20260928 05:17:58.454175 8178 fs_manager.cc:359] Metadata directory not provided
I20260928 05:17:58.454264 8178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:17:58.454545 8178 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:17:58.457340 8178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/minicluster-data/ts-1/data/instance:
uuid: "eeb06b1625b4449da5c75ca573e0f3e6"
format_stamp: "Formatted at 2026-09-28 05:17:58 on dist-test-slave-whsn"
server_key: "47b594da21489e7382d59dfa7288b240"
server_key_iv: "5d9942799d1738f0a6e7543004325a1f"
server_key_version: "encryptionkey@0"
I20260928 05:17:58.458041 8178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/minicluster-data/ts-1/wal/instance:
uuid: "eeb06b1625b4449da5c75ca573e0f3e6"
format_stamp: "Formatted at 2026-09-28 05:17:58 on dist-test-slave-whsn"
server_key: "47b594da21489e7382d59dfa7288b240"
server_key_iv: "5d9942799d1738f0a6e7543004325a1f"
server_key_version: "encryptionkey@0"
I20260928 05:17:58.462622 8178 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 05:17:58.465735 8193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:58.466938 8178 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:17:58.467073 8178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/minicluster-data/ts-1/wal
uuid: "eeb06b1625b4449da5c75ca573e0f3e6"
format_stamp: "Formatted at 2026-09-28 05:17:58 on dist-test-slave-whsn"
server_key: "47b594da21489e7382d59dfa7288b240"
server_key_iv: "5d9942799d1738f0a6e7543004325a1f"
server_key_version: "encryptionkey@0"
I20260928 05:17:58.467238 8178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:17:58.480554 8178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:17:58.483888 8178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:17:58.484099 8178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:17:58.486501 8178 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:17:58.486574 8178 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:58.486706 8178 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:17:58.486801 8178 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Sep 28 05:17:58 dist-test-slave-whsn krb5kdc[7922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572678, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260928 05:17:58.509573 8061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:58.486890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59851 (local address 127.0.97.62:39261)
0928 05:17:58.487071 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:58.487077 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:58.488310 (+ 1233us) server_negotiation.cc:407] Connection header received
0928 05:17:58.490271 (+ 1961us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:58.490277 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:58.490361 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:58.490467 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:58.492355 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:58.493356 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:58.495840 (+ 2484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:58.496470 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:58.496588 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:58.500976 (+ 4388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:58.500983 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:58.500988 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:58.501028 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:58.504335 (+ 3307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:58.505257 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:58.505262 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:58.505266 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:58.505427 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:58.505880 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:58.505884 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:58.505886 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:58.506191 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:58.506343 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:58.509048 (+ 2705us) server_negotiation.cc:299] Negotiation successful
0928 05:17:58.509251 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 05:17:58.510036 8202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:58.487370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39261 (local address 127.0.97.2:59851)
0928 05:17:58.488140 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:58.488180 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:58.490013 (+ 1833us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:58.490709 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:58.490718 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:58.491355 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:17:58.492130 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:58.492156 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:58.493559 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:58.493563 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:58.495668 (+ 2105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:58.495682 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:58.496910 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:58.496917 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:58.497179 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:58.497813 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:17:58.497837 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:17:58.500698 (+ 2861us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:17:58.504484 (+ 3786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:58.504496 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:58.504510 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:58.505122 (+ 612us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:58.505548 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:58.505553 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:58.505557 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:58.505756 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:58.506406 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:17:58.506413 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:17:58.506823 (+ 410us) client_negotiation.cc:769] Sending connection context
0928 05:17:58.507082 (+ 259us) client_negotiation.cc:240] Negotiation successful
0928 05:17:58.507557 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":321,"threads_started":1}
I20260928 05:17:58.535578 8061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:58.522684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52831 (local address 127.0.97.62:39261)
0928 05:17:58.522862 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:58.522868 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:58.523531 (+ 663us) server_negotiation.cc:407] Connection header received
0928 05:17:58.523676 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:58.523681 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:58.523749 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:58.523822 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:58.524657 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:58.525559 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:58.527376 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:58.527887 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:58.528018 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:58.530063 (+ 2045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:58.530067 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:58.530070 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:58.530096 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:58.533024 (+ 2928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:58.533618 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:58.533623 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:58.533627 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:58.533761 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:58.534120 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:58.534125 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:58.534129 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:58.534530 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:58.534681 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:58.535070 (+ 389us) server_negotiation.cc:299] Negotiation successful
0928 05:17:58.535256 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 05:17:58.535789 8266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:58.522775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39261 (local address 127.0.97.2:52831)
0928 05:17:58.523434 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:58.523450 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:58.523568 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:58.523858 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:58.523862 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:58.524266 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:17:58.524519 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:58.524527 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:58.525683 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:58.525688 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:58.527205 (+ 1517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:58.527216 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:58.528084 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:58.528091 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:58.528269 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:58.528824 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:17:58.528846 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:17:58.529870 (+ 1024us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:17:58.533127 (+ 3257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:58.533131 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:58.533134 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:58.533506 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:58.533869 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:58.533872 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:58.533875 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:58.534008 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:58.534661 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:17:58.534669 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:17:58.534829 (+ 160us) client_negotiation.cc:769] Sending connection context
0928 05:17:58.534957 (+ 128us) client_negotiation.cc:240] Negotiation successful
0928 05:17:58.535309 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":424,"threads_started":1}
I20260928 05:17:58.543923 8178 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44359
I20260928 05:17:58.543944 8312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44359 every 8 connection(s)
I20260928 05:17:58.545133 8178 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/minicluster-data/ts-1/data/info.pb
I20260928 05:17:58.546789 8313 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39261
I20260928 05:17:58.546983 8313 heartbeater.cc:464] Registering TS with master...
I20260928 05:17:58.547551 8313 heartbeater.cc:511] Master 127.0.97.62:39261 requested a full tablet report, sending...
I20260928 05:17:58.548709 7966 ts_manager.cc:194] Registered new tserver with Master: eeb06b1625b4449da5c75ca573e0f3e6 (127.0.97.2:44359)
I20260928 05:17:58.549580 7966 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:59851
I20260928 05:17:58.551585 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 8178
I20260928 05:17:58.551738 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/minicluster-data/ts-1/wal/instance
I20260928 05:17:58.552121 388 external_mini_cluster.cc:1428] Setting key 6d9fbef00b62b459a8ffb7d058a2986a
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260928 05:17:58.591893 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39261
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:17:58.887215 8317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:17:58.887457 8317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:17:58.887583 8317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:17:58.894375 8317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:17:58.894471 8317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:17:58.894610 8317 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 05:17:58.894708 8317 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 05:17:58.903955 8317 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:39261
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:17:58.905215 8317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:17:58.906430 8317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:17:58.918299 8322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:58.918359 8325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:17:58.918340 8323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:17:58.919049 8317 server_base.cc:1034] running on GCE node
I20260928 05:17:58.919525 8317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:17:58.920276 8317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:17:58.921463 8317 hybrid_clock.cc:648] HybridClock initialized: now 1790572678921403 us; error 40 us; skew 500 ppm
I20260928 05:17:58.921720 8317 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:17:58 dist-test-slave-whsn krb5kdc[7922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572678, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:17:58.928457 8317 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260928 05:17:58.929774 8317 webserver.cc:460] Webserver started at http://127.0.97.3:42555/ using document root <none> and password file <none>
I20260928 05:17:58.930353 8317 fs_manager.cc:359] Metadata directory not provided
I20260928 05:17:58.930429 8317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:17:58.930677 8317 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:17:58.933337 8317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/minicluster-data/ts-2/data/instance:
uuid: "c35d4d4f8f3a49a09bfaec1f37147dd2"
format_stamp: "Formatted at 2026-09-28 05:17:58 on dist-test-slave-whsn"
server_key: "da4af2fde290395a91d794976b6210df"
server_key_iv: "15be8834aed252bb1e87fa5dcfa8c0ba"
server_key_version: "encryptionkey@0"
I20260928 05:17:58.934043 8317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/minicluster-data/ts-2/wal/instance:
uuid: "c35d4d4f8f3a49a09bfaec1f37147dd2"
format_stamp: "Formatted at 2026-09-28 05:17:58 on dist-test-slave-whsn"
server_key: "da4af2fde290395a91d794976b6210df"
server_key_iv: "15be8834aed252bb1e87fa5dcfa8c0ba"
server_key_version: "encryptionkey@0"
I20260928 05:17:58.938624 8317 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 05:17:58.941821 8332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:58.943027 8317 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:17:58.943177 8317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/minicluster-data/ts-2/wal
uuid: "c35d4d4f8f3a49a09bfaec1f37147dd2"
format_stamp: "Formatted at 2026-09-28 05:17:58 on dist-test-slave-whsn"
server_key: "da4af2fde290395a91d794976b6210df"
server_key_iv: "15be8834aed252bb1e87fa5dcfa8c0ba"
server_key_version: "encryptionkey@0"
I20260928 05:17:58.943357 8317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:17:58.959714 8317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:17:58.963090 8317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:17:58.963312 8317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:17:58.965843 8317 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:17:58.965937 8317 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:17:58.966060 8317 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:17:58.966127 8317 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 05:17:58 dist-test-slave-whsn krb5kdc[7922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572678, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260928 05:17:58.987165 8061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:58.966345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41073 (local address 127.0.97.62:39261)
0928 05:17:58.966549 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:58.966555 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:58.967831 (+ 1276us) server_negotiation.cc:407] Connection header received
0928 05:17:58.969808 (+ 1977us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:58.969813 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:58.969894 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:58.969981 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:58.972116 (+ 2135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:58.973113 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:58.975371 (+ 2258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:58.975869 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:58.975983 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:58.980199 (+ 4216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:58.980207 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:58.980211 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:58.980251 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:58.983605 (+ 3354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:58.984440 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:58.984446 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:58.984451 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:58.984610 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:58.985106 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:58.985112 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:58.985114 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:58.985461 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:58.985598 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:58.986529 (+ 931us) server_negotiation.cc:299] Negotiation successful
0928 05:17:58.986767 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 05:17:58.988065 8341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:58.966946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39261 (local address 127.0.97.3:41073)
0928 05:17:58.967687 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:58.967720 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:58.969524 (+ 1804us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:58.970288 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:58.970303 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:58.971084 (+ 781us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:17:58.971933 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:58.971970 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:58.973269 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:58.973274 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:58.975239 (+ 1965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:58.975251 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:58.976091 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:58.976099 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:58.976294 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:58.977013 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:17:58.977034 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:17:58.979986 (+ 2952us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:17:58.983738 (+ 3752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:58.983770 (+ 32us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:58.983787 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:58.984325 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:58.984725 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:58.984728 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:58.984730 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:58.984986 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:58.985590 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:17:58.985597 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:17:58.986074 (+ 477us) client_negotiation.cc:769] Sending connection context
0928 05:17:58.986406 (+ 332us) client_negotiation.cc:240] Negotiation successful
0928 05:17:58.986778 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":301,"threads_started":1}
I20260928 05:17:59.006498 8061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:58.993303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34069 (local address 127.0.97.62:39261)
0928 05:17:58.993463 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:58.993468 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:58.994141 (+ 673us) server_negotiation.cc:407] Connection header received
0928 05:17:58.994289 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:58.994295 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:58.994363 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:58.994447 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:58.995839 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:58.996564 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:58.998674 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:58.999094 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:58.999185 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:59.001285 (+ 2100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:59.001289 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:59.001292 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:59.001320 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:59.003657 (+ 2337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:59.004411 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:59.004414 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:59.004416 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:59.004534 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:59.004995 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:59.005001 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:59.005004 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:59.005346 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:59.005502 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:59.005937 (+ 435us) server_negotiation.cc:299] Negotiation successful
0928 05:17:59.006159 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 05:17:59.006613 8403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:58.993393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39261 (local address 127.0.97.3:34069)
0928 05:17:58.994049 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:58.994064 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:58.994185 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:58.994525 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:58.994530 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:58.995180 (+ 650us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:17:58.995679 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:58.995693 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:58.996677 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:58.996683 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:58.998530 (+ 1847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:58.998544 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:58.999271 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:58.999278 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:58.999447 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:59.000031 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:17:59.000069 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:17:59.001111 (+ 1042us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:17:59.003770 (+ 2659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:59.003776 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:59.003780 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:59.004304 (+ 524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:59.004671 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:59.004677 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:59.004681 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:59.004841 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:59.005485 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:17:59.005491 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:17:59.005672 (+ 181us) client_negotiation.cc:769] Sending connection context
0928 05:17:59.005794 (+ 122us) client_negotiation.cc:240] Negotiation successful
0928 05:17:59.006141 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":394,"threads_started":1}
I20260928 05:17:59.015163 8317 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35975
I20260928 05:17:59.015184 8451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35975 every 8 connection(s)
I20260928 05:17:59.016273 8317 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/minicluster-data/ts-2/data/info.pb
I20260928 05:17:59.017565 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 8317
I20260928 05:17:59.017709 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/minicluster-data/ts-2/wal/instance
I20260928 05:17:59.018074 388 external_mini_cluster.cc:1428] Setting key f060d8d7c8ba1370bbfdbebd41483af5
I20260928 05:17:59.019218 8452 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39261
I20260928 05:17:59.019436 8452 heartbeater.cc:464] Registering TS with master...
I20260928 05:17:59.020128 8452 heartbeater.cc:511] Master 127.0.97.62:39261 requested a full tablet report, sending...
I20260928 05:17:59.021339 7967 ts_manager.cc:194] Registered new tserver with Master: c35d4d4f8f3a49a09bfaec1f37147dd2 (127.0.97.3:35975)
I20260928 05:17:59.022071 7967 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:41073
I20260928 05:17:59.032593 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 05:17:59 dist-test-slave-whsn krb5kdc[7922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572679, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 05:17:59 dist-test-slave-whsn krb5kdc[7922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:17:59 dist-test-slave-whsn krb5kdc[7922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 05:17:59.075773 8061 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:17:59.065623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53240 (local address 127.0.97.62:39261)
0928 05:17:59.065845 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:59.065852 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:59.066084 (+ 232us) server_negotiation.cc:407] Connection header received
0928 05:17:59.066277 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:59.066284 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:59.066352 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:59.066453 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:59.067767 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:59.068755 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:59.069982 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:59.070632 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:59.070767 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:59.075573 (+ 4806us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53240: BlockingRecv error: recv got EOF from 127.0.0.1:53240 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 05:17:59.082343 8174 heartbeater.cc:503] Master 127.0.97.62:39261 was elected leader, sending a full tablet report...
Sep 28 05:17:59 dist-test-slave-whsn krb5kdc[7922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572679, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260928 05:17:59.090528 8061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:59.078469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53242 (local address 127.0.97.62:39261)
0928 05:17:59.078609 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:59.078612 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:59.078883 (+ 271us) server_negotiation.cc:407] Connection header received
0928 05:17:59.078997 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:59.079001 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:59.079066 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:59.079138 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:59.080119 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:59.080772 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:59.081944 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:59.082289 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:59.082370 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:59.086074 (+ 3704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:59.086079 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:59.086082 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:59.086109 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:59.088433 (+ 2324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:59.089012 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:59.089016 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:59.089019 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:59.089119 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:59.089493 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:59.089496 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:59.089497 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:59.089733 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:59.089927 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:59.090145 (+ 218us) server_negotiation.cc:299] Negotiation successful
0928 05:17:59.090287 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 05:17:59.093899 7967 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53242:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 05:17:59.100256 7967 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 05:17:59.135087 8471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:59.114245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47436 (local address 127.0.97.2:44359)
0928 05:17:59.127712 (+ 13467us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:59.127721 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:59.127753 (+ 32us) server_negotiation.cc:407] Connection header received
0928 05:17:59.127857 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:59.127866 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:59.128169 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:59.128368 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:17:59.129096 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:59.130515 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:59.132483 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:59.133872 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:59.134004 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:59.134248 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:59.134541 (+ 293us) server_negotiation.cc:299] Negotiation successful
0928 05:17:59.134698 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13302,"thread_start_us":13189,"threads_started":1}
I20260928 05:17:59.135462 8473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:59.114966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43410 (local address 127.0.97.3:35975)
0928 05:17:59.127456 (+ 12490us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:59.127464 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:59.127935 (+ 471us) server_negotiation.cc:407] Connection header received
0928 05:17:59.128086 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:59.128093 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:59.128335 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:59.128511 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:17:59.129490 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:59.130709 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:59.133187 (+ 2478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:59.134135 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:59.134275 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:59.134482 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:59.134984 (+ 502us) server_negotiation.cc:299] Negotiation successful
0928 05:17:59.135152 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12307,"thread_start_us":12146,"threads_started":1}
I20260928 05:17:59.136044 8470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:59.114130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44359 (local address 127.0.0.1:47436)
0928 05:17:59.122114 (+ 7984us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:59.122155 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:59.122344 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:59.128395 (+ 6051us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:59.128402 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:59.128432 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:17:59.128901 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:59.128914 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:59.130624 (+ 1710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:59.130629 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:59.132305 (+ 1676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:59.132318 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:59.134062 (+ 1744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:59.134069 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:59.134311 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:59.134363 (+ 52us) client_negotiation.cc:769] Sending connection context
0928 05:17:59.134489 (+ 126us) client_negotiation.cc:240] Negotiation successful
0928 05:17:59.134674 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1926,"mutex_wait_us":32,"thread_start_us":1005,"threads_started":1}
I20260928 05:17:59.136127 8475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:59.114452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35975 (local address 127.0.0.1:43410)
0928 05:17:59.127828 (+ 13376us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:59.127841 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:59.127949 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:59.128780 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:59.128787 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:59.128812 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:17:59.129264 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:59.129275 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:59.130883 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:59.130888 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:59.132975 (+ 2087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:59.132987 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:59.134638 (+ 1651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:59.134644 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:59.134779 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:59.134814 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 05:17:59.135653 (+ 839us) client_negotiation.cc:240] Negotiation successful
0928 05:17:59.135770 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13286,"mutex_wait_us":43,"thread_start_us":14211,"threads_started":1}
I20260928 05:17:59.136777 8474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:59.114733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42423 (local address 127.0.0.1:46714)
0928 05:17:59.128984 (+ 14251us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:59.128998 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:59.129193 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:59.129968 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:59.129972 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:59.129996 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:17:59.130310 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:59.130319 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:59.132032 (+ 1713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:59.132037 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:59.134126 (+ 2089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:59.134139 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:59.136020 (+ 1881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:59.136027 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:59.136200 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:59.136233 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 05:17:59.136311 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 05:17:59.136419 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14101,"mutex_wait_us":1338,"thread_start_us":11945,"threads_started":1}
I20260928 05:17:59.137391 8472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:59.114953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46714 (local address 127.0.97.1:42423)
0928 05:17:59.115461 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:59.115483 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:59.129086 (+ 13603us) server_negotiation.cc:407] Connection header received
0928 05:17:59.129377 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:59.129385 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:59.129585 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:59.129735 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:17:59.130452 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:59.131893 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:59.134296 (+ 2403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:59.135583 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:59.135700 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:59.135875 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:59.136981 (+ 1106us) server_negotiation.cc:299] Negotiation successful
0928 05:17:59.137132 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":243,"threads_started":1}
I20260928 05:17:59.139983 8382 tablet_service.cc:1467] Processing CreateTablet for tablet 2144e0091fff456a8ecdef1199a22507 (DEFAULT_TABLE table=test-table [id=e97c41786c094560a03e56916e9fab88]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:17:59.140486 8104 tablet_service.cc:1467] Processing CreateTablet for tablet 2144e0091fff456a8ecdef1199a22507 (DEFAULT_TABLE table=test-table [id=e97c41786c094560a03e56916e9fab88]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:17:59.141453 8382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2144e0091fff456a8ecdef1199a22507. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:17:59.141696 8104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2144e0091fff456a8ecdef1199a22507. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:17:59.142704 8243 tablet_service.cc:1467] Processing CreateTablet for tablet 2144e0091fff456a8ecdef1199a22507 (DEFAULT_TABLE table=test-table [id=e97c41786c094560a03e56916e9fab88]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:17:59.143563 8243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2144e0091fff456a8ecdef1199a22507. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:17:59.153529 8476 tablet_bootstrap.cc:492] T 2144e0091fff456a8ecdef1199a22507 P eeb06b1625b4449da5c75ca573e0f3e6: Bootstrap starting.
I20260928 05:17:59.155303 8478 tablet_bootstrap.cc:492] T 2144e0091fff456a8ecdef1199a22507 P c35d4d4f8f3a49a09bfaec1f37147dd2: Bootstrap starting.
I20260928 05:17:59.157215 8476 tablet_bootstrap.cc:654] T 2144e0091fff456a8ecdef1199a22507 P eeb06b1625b4449da5c75ca573e0f3e6: Neither blocks nor log segments found. Creating new log.
I20260928 05:17:59.157464 8477 tablet_bootstrap.cc:492] T 2144e0091fff456a8ecdef1199a22507 P 0f89f414d76944cfb5a9505edaa66d35: Bootstrap starting.
I20260928 05:17:59.158413 8478 tablet_bootstrap.cc:654] T 2144e0091fff456a8ecdef1199a22507 P c35d4d4f8f3a49a09bfaec1f37147dd2: Neither blocks nor log segments found. Creating new log.
I20260928 05:17:59.158484 8476 log.cc:824] T 2144e0091fff456a8ecdef1199a22507 P eeb06b1625b4449da5c75ca573e0f3e6: Log is configured to *not* fsync() on all Append() calls
I20260928 05:17:59.159642 8478 log.cc:824] T 2144e0091fff456a8ecdef1199a22507 P c35d4d4f8f3a49a09bfaec1f37147dd2: Log is configured to *not* fsync() on all Append() calls
I20260928 05:17:59.160375 8477 tablet_bootstrap.cc:654] T 2144e0091fff456a8ecdef1199a22507 P 0f89f414d76944cfb5a9505edaa66d35: Neither blocks nor log segments found. Creating new log.
I20260928 05:17:59.161371 8477 log.cc:824] T 2144e0091fff456a8ecdef1199a22507 P 0f89f414d76944cfb5a9505edaa66d35: Log is configured to *not* fsync() on all Append() calls
I20260928 05:17:59.162479 8478 tablet_bootstrap.cc:492] T 2144e0091fff456a8ecdef1199a22507 P c35d4d4f8f3a49a09bfaec1f37147dd2: No bootstrap required, opened a new log
I20260928 05:17:59.162470 8476 tablet_bootstrap.cc:492] T 2144e0091fff456a8ecdef1199a22507 P eeb06b1625b4449da5c75ca573e0f3e6: No bootstrap required, opened a new log
I20260928 05:17:59.162707 8478 ts_tablet_manager.cc:1397] T 2144e0091fff456a8ecdef1199a22507 P c35d4d4f8f3a49a09bfaec1f37147dd2: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260928 05:17:59.162709 8476 ts_tablet_manager.cc:1397] T 2144e0091fff456a8ecdef1199a22507 P eeb06b1625b4449da5c75ca573e0f3e6: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.004s
I20260928 05:17:59.163524 8477 tablet_bootstrap.cc:492] T 2144e0091fff456a8ecdef1199a22507 P 0f89f414d76944cfb5a9505edaa66d35: No bootstrap required, opened a new log
I20260928 05:17:59.163741 8477 ts_tablet_manager.cc:1397] T 2144e0091fff456a8ecdef1199a22507 P 0f89f414d76944cfb5a9505edaa66d35: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260928 05:17:59.170483 8477 raft_consensus.cc:348] T 2144e0091fff456a8ecdef1199a22507 P 0f89f414d76944cfb5a9505edaa66d35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb06b1625b4449da5c75ca573e0f3e6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44359 } } peers { permanent_uuid: "c35d4d4f8f3a49a09bfaec1f37147dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35975 } } peers { permanent_uuid: "0f89f414d76944cfb5a9505edaa66d35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42423 } }
I20260928 05:17:59.170780 8477 raft_consensus.cc:374] T 2144e0091fff456a8ecdef1199a22507 P 0f89f414d76944cfb5a9505edaa66d35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:17:59.170871 8477 raft_consensus.cc:729] T 2144e0091fff456a8ecdef1199a22507 P 0f89f414d76944cfb5a9505edaa66d35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f89f414d76944cfb5a9505edaa66d35, State: Initialized, Role: FOLLOWER
I20260928 05:17:59.171415 8477 consensus_queue.cc:260] T 2144e0091fff456a8ecdef1199a22507 P 0f89f414d76944cfb5a9505edaa66d35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb06b1625b4449da5c75ca573e0f3e6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44359 } } peers { permanent_uuid: "c35d4d4f8f3a49a09bfaec1f37147dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35975 } } peers { permanent_uuid: "0f89f414d76944cfb5a9505edaa66d35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42423 } }
I20260928 05:17:59.172544 8476 raft_consensus.cc:348] T 2144e0091fff456a8ecdef1199a22507 P eeb06b1625b4449da5c75ca573e0f3e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb06b1625b4449da5c75ca573e0f3e6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44359 } } peers { permanent_uuid: "c35d4d4f8f3a49a09bfaec1f37147dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35975 } } peers { permanent_uuid: "0f89f414d76944cfb5a9505edaa66d35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42423 } }
I20260928 05:17:59.172561 8478 raft_consensus.cc:348] T 2144e0091fff456a8ecdef1199a22507 P c35d4d4f8f3a49a09bfaec1f37147dd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb06b1625b4449da5c75ca573e0f3e6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44359 } } peers { permanent_uuid: "c35d4d4f8f3a49a09bfaec1f37147dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35975 } } peers { permanent_uuid: "0f89f414d76944cfb5a9505edaa66d35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42423 } }
I20260928 05:17:59.172928 8478 raft_consensus.cc:374] T 2144e0091fff456a8ecdef1199a22507 P c35d4d4f8f3a49a09bfaec1f37147dd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:17:59.172945 8476 raft_consensus.cc:374] T 2144e0091fff456a8ecdef1199a22507 P eeb06b1625b4449da5c75ca573e0f3e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:17:59.173024 8478 raft_consensus.cc:729] T 2144e0091fff456a8ecdef1199a22507 P c35d4d4f8f3a49a09bfaec1f37147dd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c35d4d4f8f3a49a09bfaec1f37147dd2, State: Initialized, Role: FOLLOWER
I20260928 05:17:59.173024 8476 raft_consensus.cc:729] T 2144e0091fff456a8ecdef1199a22507 P eeb06b1625b4449da5c75ca573e0f3e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeb06b1625b4449da5c75ca573e0f3e6, State: Initialized, Role: FOLLOWER
I20260928 05:17:59.173830 8477 ts_tablet_manager.cc:1428] T 2144e0091fff456a8ecdef1199a22507 P 0f89f414d76944cfb5a9505edaa66d35: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260928 05:17:59.173676 8476 consensus_queue.cc:260] T 2144e0091fff456a8ecdef1199a22507 P eeb06b1625b4449da5c75ca573e0f3e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb06b1625b4449da5c75ca573e0f3e6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44359 } } peers { permanent_uuid: "c35d4d4f8f3a49a09bfaec1f37147dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35975 } } peers { permanent_uuid: "0f89f414d76944cfb5a9505edaa66d35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42423 } }
I20260928 05:17:59.173717 8478 consensus_queue.cc:260] T 2144e0091fff456a8ecdef1199a22507 P c35d4d4f8f3a49a09bfaec1f37147dd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb06b1625b4449da5c75ca573e0f3e6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44359 } } peers { permanent_uuid: "c35d4d4f8f3a49a09bfaec1f37147dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35975 } } peers { permanent_uuid: "0f89f414d76944cfb5a9505edaa66d35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42423 } }
I20260928 05:17:59.180599 8313 heartbeater.cc:503] Master 127.0.97.62:39261 was elected leader, sending a full tablet report...
I20260928 05:17:59.180670 8452 heartbeater.cc:503] Master 127.0.97.62:39261 was elected leader, sending a full tablet report...
I20260928 05:17:59.181494 8476 ts_tablet_manager.cc:1428] T 2144e0091fff456a8ecdef1199a22507 P eeb06b1625b4449da5c75ca573e0f3e6: Time spent starting tablet: real 0.019s user 0.008s sys 0.011s
I20260928 05:17:59.181520 8478 ts_tablet_manager.cc:1428] T 2144e0091fff456a8ecdef1199a22507 P c35d4d4f8f3a49a09bfaec1f37147dd2: Time spent starting tablet: real 0.019s user 0.019s sys 0.000s
W20260928 05:17:59.267650 8453 tablet.cc:2369] T 2144e0091fff456a8ecdef1199a22507 P c35d4d4f8f3a49a09bfaec1f37147dd2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 05:17:59.296779 8314 tablet.cc:2369] T 2144e0091fff456a8ecdef1199a22507 P eeb06b1625b4449da5c75ca573e0f3e6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 05:17:59.322397 8175 tablet.cc:2369] T 2144e0091fff456a8ecdef1199a22507 P 0f89f414d76944cfb5a9505edaa66d35: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 05:17:59.427299 8483 raft_consensus.cc:482] T 2144e0091fff456a8ecdef1199a22507 P eeb06b1625b4449da5c75ca573e0f3e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:17:59.427592 8483 raft_consensus.cc:504] T 2144e0091fff456a8ecdef1199a22507 P eeb06b1625b4449da5c75ca573e0f3e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb06b1625b4449da5c75ca573e0f3e6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44359 } } peers { permanent_uuid: "c35d4d4f8f3a49a09bfaec1f37147dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35975 } } peers { permanent_uuid: "0f89f414d76944cfb5a9505edaa66d35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42423 } }
I20260928 05:17:59.429216 8483 leader_election.cc:288] T 2144e0091fff456a8ecdef1199a22507 P eeb06b1625b4449da5c75ca573e0f3e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c35d4d4f8f3a49a09bfaec1f37147dd2 (127.0.97.3:35975), 0f89f414d76944cfb5a9505edaa66d35 (127.0.97.1:42423)
I20260928 05:17:59.437562 8473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:59.429585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43831 (local address 127.0.97.3:35975)
0928 05:17:59.429761 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:59.429767 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:59.430554 (+ 787us) server_negotiation.cc:407] Connection header received
0928 05:17:59.430705 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:59.430710 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:59.430785 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:59.430885 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:17:59.431570 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:59.432615 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:59.435378 (+ 2763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:59.436524 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:59.436653 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:59.436754 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:59.437195 (+ 441us) server_negotiation.cc:299] Negotiation successful
0928 05:17:59.437319 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 05:17:59.437554 8472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:59.429648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56749 (local address 127.0.97.1:42423)
0928 05:17:59.429811 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:59.429817 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:59.430533 (+ 716us) server_negotiation.cc:407] Connection header received
0928 05:17:59.430726 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:59.430731 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:59.430806 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:59.430884 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:17:59.431577 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:59.432613 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:59.435368 (+ 2755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:59.436524 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:59.436678 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:59.436763 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:59.437196 (+ 433us) server_negotiation.cc:299] Negotiation successful
0928 05:17:59.437319 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 05:17:59.437803 8485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:59.429498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35975 (local address 127.0.97.2:43831)
0928 05:17:59.430452 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:59.430465 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:59.430581 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:59.430907 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:59.430911 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:59.430940 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:17:59.431400 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:59.431414 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:59.432756 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:59.432762 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:59.435223 (+ 2461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:59.435235 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:59.436774 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:59.436785 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:59.436999 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:59.437076 (+ 77us) client_negotiation.cc:769] Sending connection context
0928 05:17:59.437169 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 05:17:59.437362 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":837,"mutex_wait_us":183,"thread_start_us":401,"threads_started":1}
I20260928 05:17:59.438094 8486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:59.429449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42423 (local address 127.0.97.2:56749)
0928 05:17:59.430403 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:59.430431 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:59.430616 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:59.430907 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:59.430911 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:59.430971 (+ 60us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:17:59.431472 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:59.431481 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:59.432774 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:59.432779 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:59.435221 (+ 2442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:59.435235 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:59.436792 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:59.436797 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:59.437066 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:59.437092 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 05:17:59.437169 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 05:17:59.437362 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":820,"mutex_wait_us":173,"spinlock_wait_cycles":36480,"thread_start_us":722,"threads_started":1}
I20260928 05:17:59.438714 8407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2144e0091fff456a8ecdef1199a22507" candidate_uuid: "eeb06b1625b4449da5c75ca573e0f3e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c35d4d4f8f3a49a09bfaec1f37147dd2" is_pre_election: true
I20260928 05:17:59.438968 8124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2144e0091fff456a8ecdef1199a22507" candidate_uuid: "eeb06b1625b4449da5c75ca573e0f3e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f89f414d76944cfb5a9505edaa66d35" is_pre_election: true
I20260928 05:17:59.439281 8407 raft_consensus.cc:2454] T 2144e0091fff456a8ecdef1199a22507 P c35d4d4f8f3a49a09bfaec1f37147dd2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeb06b1625b4449da5c75ca573e0f3e6 in term 0.
I20260928 05:17:59.439435 8124 raft_consensus.cc:2454] T 2144e0091fff456a8ecdef1199a22507 P 0f89f414d76944cfb5a9505edaa66d35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeb06b1625b4449da5c75ca573e0f3e6 in term 0.
I20260928 05:17:59.439949 8196 leader_election.cc:302] T 2144e0091fff456a8ecdef1199a22507 P eeb06b1625b4449da5c75ca573e0f3e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0f89f414d76944cfb5a9505edaa66d35, c35d4d4f8f3a49a09bfaec1f37147dd2, eeb06b1625b4449da5c75ca573e0f3e6; no voters:
I20260928 05:17:59.440320 8483 raft_consensus.cc:2781] T 2144e0091fff456a8ecdef1199a22507 P eeb06b1625b4449da5c75ca573e0f3e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 05:17:59.440459 8483 raft_consensus.cc:482] T 2144e0091fff456a8ecdef1199a22507 P eeb06b1625b4449da5c75ca573e0f3e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 05:17:59.440557 8483 raft_consensus.cc:3037] T 2144e0091fff456a8ecdef1199a22507 P eeb06b1625b4449da5c75ca573e0f3e6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:17:59.442698 8483 raft_consensus.cc:504] T 2144e0091fff456a8ecdef1199a22507 P eeb06b1625b4449da5c75ca573e0f3e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb06b1625b4449da5c75ca573e0f3e6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44359 } } peers { permanent_uuid: "c35d4d4f8f3a49a09bfaec1f37147dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35975 } } peers { permanent_uuid: "0f89f414d76944cfb5a9505edaa66d35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42423 } }
I20260928 05:17:59.443205 8483 leader_election.cc:288] T 2144e0091fff456a8ecdef1199a22507 P eeb06b1625b4449da5c75ca573e0f3e6 [CANDIDATE]: Term 1 election: Requested vote from peers c35d4d4f8f3a49a09bfaec1f37147dd2 (127.0.97.3:35975), 0f89f414d76944cfb5a9505edaa66d35 (127.0.97.1:42423)
I20260928 05:17:59.443554 8407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2144e0091fff456a8ecdef1199a22507" candidate_uuid: "eeb06b1625b4449da5c75ca573e0f3e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c35d4d4f8f3a49a09bfaec1f37147dd2"
I20260928 05:17:59.443620 8124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2144e0091fff456a8ecdef1199a22507" candidate_uuid: "eeb06b1625b4449da5c75ca573e0f3e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f89f414d76944cfb5a9505edaa66d35"
I20260928 05:17:59.443744 8407 raft_consensus.cc:3037] T 2144e0091fff456a8ecdef1199a22507 P c35d4d4f8f3a49a09bfaec1f37147dd2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:17:59.443768 8124 raft_consensus.cc:3037] T 2144e0091fff456a8ecdef1199a22507 P 0f89f414d76944cfb5a9505edaa66d35 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:17:59.446717 8407 raft_consensus.cc:2454] T 2144e0091fff456a8ecdef1199a22507 P c35d4d4f8f3a49a09bfaec1f37147dd2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeb06b1625b4449da5c75ca573e0f3e6 in term 1.
I20260928 05:17:59.446717 8124 raft_consensus.cc:2454] T 2144e0091fff456a8ecdef1199a22507 P 0f89f414d76944cfb5a9505edaa66d35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeb06b1625b4449da5c75ca573e0f3e6 in term 1.
I20260928 05:17:59.447093 8197 leader_election.cc:302] T 2144e0091fff456a8ecdef1199a22507 P eeb06b1625b4449da5c75ca573e0f3e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0f89f414d76944cfb5a9505edaa66d35, c35d4d4f8f3a49a09bfaec1f37147dd2, eeb06b1625b4449da5c75ca573e0f3e6; no voters:
I20260928 05:17:59.447371 8483 raft_consensus.cc:2781] T 2144e0091fff456a8ecdef1199a22507 P eeb06b1625b4449da5c75ca573e0f3e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:17:59.447780 8483 raft_consensus.cc:686] T 2144e0091fff456a8ecdef1199a22507 P eeb06b1625b4449da5c75ca573e0f3e6 [term 1 LEADER]: Becoming Leader. State: Replica: eeb06b1625b4449da5c75ca573e0f3e6, State: Running, Role: LEADER
I20260928 05:17:59.448236 8483 consensus_queue.cc:237] T 2144e0091fff456a8ecdef1199a22507 P eeb06b1625b4449da5c75ca573e0f3e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb06b1625b4449da5c75ca573e0f3e6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44359 } } peers { permanent_uuid: "c35d4d4f8f3a49a09bfaec1f37147dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35975 } } peers { permanent_uuid: "0f89f414d76944cfb5a9505edaa66d35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42423 } }
I20260928 05:17:59.452153 7967 catalog_manager.cc:5495] T 2144e0091fff456a8ecdef1199a22507 P eeb06b1625b4449da5c75ca573e0f3e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to eeb06b1625b4449da5c75ca573e0f3e6 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "eeb06b1625b4449da5c75ca573e0f3e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb06b1625b4449da5c75ca573e0f3e6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c35d4d4f8f3a49a09bfaec1f37147dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f89f414d76944cfb5a9505edaa66d35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42423 } health_report { overall_health: UNKNOWN } } }
I20260928 05:17:59.500414 8471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:59.495030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47452 (local address 127.0.97.2:44359)
0928 05:17:59.495227 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:59.495233 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:59.495335 (+ 102us) server_negotiation.cc:407] Connection header received
0928 05:17:59.495516 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:59.495519 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:59.495590 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:59.495705 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:17:59.496033 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:59.496711 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:59.498548 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:59.498918 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:59.499001 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:59.499457 (+ 456us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:17:59.499941 (+ 484us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:17:59.500053 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:59.500145 (+ 92us) server_negotiation.cc:299] Negotiation successful
0928 05:17:59.500203 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 05:17:59.508359 8407 raft_consensus.cc:1261] T 2144e0091fff456a8ecdef1199a22507 P c35d4d4f8f3a49a09bfaec1f37147dd2 [term 1 FOLLOWER]: Refusing update from remote peer eeb06b1625b4449da5c75ca573e0f3e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:17:59.508478 8124 raft_consensus.cc:1261] T 2144e0091fff456a8ecdef1199a22507 P 0f89f414d76944cfb5a9505edaa66d35 [term 1 FOLLOWER]: Refusing update from remote peer eeb06b1625b4449da5c75ca573e0f3e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:17:59.509284 8483 consensus_queue.cc:1035] T 2144e0091fff456a8ecdef1199a22507 P eeb06b1625b4449da5c75ca573e0f3e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c35d4d4f8f3a49a09bfaec1f37147dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:17:59.509572 8487 consensus_queue.cc:1035] T 2144e0091fff456a8ecdef1199a22507 P eeb06b1625b4449da5c75ca573e0f3e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f89f414d76944cfb5a9505edaa66d35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:17:59.523156 8491 mvcc.cc:205] Tried to move back new op lower bound from 7334185695256694784 to 7334185695023083520. Current Snapshot: MvccSnapshot[applied={T|T < 7334185695256694784 or (T in {7334185695256694784})}]
I20260928 05:17:59.531039 7967 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53242:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:17:59.531283 7967 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53242:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:17:59.534524 7967 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ca2877e519c94fd181176dfe63838bb3: Sending DeleteTablet for 3 replicas of tablet 2144e0091fff456a8ecdef1199a22507
I20260928 05:17:59.535465 8382 tablet_service.cc:1514] Processing DeleteTablet for tablet 2144e0091fff456a8ecdef1199a22507 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:17:59 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43410
I20260928 05:17:59.535490 8243 tablet_service.cc:1514] Processing DeleteTablet for tablet 2144e0091fff456a8ecdef1199a22507 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:17:59 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47436
I20260928 05:17:59.535821 8104 tablet_service.cc:1514] Processing DeleteTablet for tablet 2144e0091fff456a8ecdef1199a22507 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:17:59 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46714
I20260928 05:17:59.537721 7967 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53242:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 05:17:59.538996 7967 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 05:17:59.541586 8502 tablet_replica.cc:330] T 2144e0091fff456a8ecdef1199a22507 P 0f89f414d76944cfb5a9505edaa66d35: stopping tablet replica
I20260928 05:17:59.541977 8502 raft_consensus.cc:2229] T 2144e0091fff456a8ecdef1199a22507 P 0f89f414d76944cfb5a9505edaa66d35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:17:59.542426 8502 raft_consensus.cc:2258] T 2144e0091fff456a8ecdef1199a22507 P 0f89f414d76944cfb5a9505edaa66d35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:17:59.544482 8502 ts_tablet_manager.cc:1905] T 2144e0091fff456a8ecdef1199a22507 P 0f89f414d76944cfb5a9505edaa66d35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:17:59.547878 8381 tablet_service.cc:1467] Processing CreateTablet for tablet f07bad35e54046afb27621bb02449472 (DEFAULT_TABLE table=test-table [id=b5792c948c2745968bc0ef8e930b78ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:17:59.548398 8381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f07bad35e54046afb27621bb02449472. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:17:59.549832 8242 tablet_service.cc:1467] Processing CreateTablet for tablet f07bad35e54046afb27621bb02449472 (DEFAULT_TABLE table=test-table [id=b5792c948c2745968bc0ef8e930b78ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:17:59.550369 8242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f07bad35e54046afb27621bb02449472. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:17:59.557646 8502 ts_tablet_manager.cc:1918] T 2144e0091fff456a8ecdef1199a22507 P 0f89f414d76944cfb5a9505edaa66d35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:17:59.557806 8502 log.cc:1196] T 2144e0091fff456a8ecdef1199a22507 P 0f89f414d76944cfb5a9505edaa66d35: Deleting WAL directory at /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/minicluster-data/ts-0/wal/wals/2144e0091fff456a8ecdef1199a22507
I20260928 05:17:59.557801 8104 tablet_service.cc:1467] Processing CreateTablet for tablet f07bad35e54046afb27621bb02449472 (DEFAULT_TABLE table=test-table [id=b5792c948c2745968bc0ef8e930b78ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:17:59.558346 8104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f07bad35e54046afb27621bb02449472. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:17:59.559580 8500 tablet_replica.cc:330] T 2144e0091fff456a8ecdef1199a22507 P c35d4d4f8f3a49a09bfaec1f37147dd2: stopping tablet replica
I20260928 05:17:59.560123 8500 raft_consensus.cc:2229] T 2144e0091fff456a8ecdef1199a22507 P c35d4d4f8f3a49a09bfaec1f37147dd2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:17:59.560585 8500 raft_consensus.cc:2258] T 2144e0091fff456a8ecdef1199a22507 P c35d4d4f8f3a49a09bfaec1f37147dd2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:17:59.561199 8502 ts_tablet_manager.cc:1939] T 2144e0091fff456a8ecdef1199a22507 P 0f89f414d76944cfb5a9505edaa66d35: Deleting consensus metadata
I20260928 05:17:59.562984 7954 catalog_manager.cc:4841] TS 0f89f414d76944cfb5a9505edaa66d35 (127.0.97.1:42423): tablet 2144e0091fff456a8ecdef1199a22507 (table test-table [id=e97c41786c094560a03e56916e9fab88]) successfully deleted
I20260928 05:17:59.564492 8500 ts_tablet_manager.cc:1905] T 2144e0091fff456a8ecdef1199a22507 P c35d4d4f8f3a49a09bfaec1f37147dd2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:17:59.569288 8501 tablet_replica.cc:330] T 2144e0091fff456a8ecdef1199a22507 P eeb06b1625b4449da5c75ca573e0f3e6: stopping tablet replica
I20260928 05:17:59.569696 8501 raft_consensus.cc:2229] T 2144e0091fff456a8ecdef1199a22507 P eeb06b1625b4449da5c75ca573e0f3e6 [term 1 LEADER]: Raft consensus shutting down.
I20260928 05:17:59.571069 8500 ts_tablet_manager.cc:1918] T 2144e0091fff456a8ecdef1199a22507 P c35d4d4f8f3a49a09bfaec1f37147dd2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:17:59.571201 8500 log.cc:1196] T 2144e0091fff456a8ecdef1199a22507 P c35d4d4f8f3a49a09bfaec1f37147dd2: Deleting WAL directory at /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/minicluster-data/ts-2/wal/wals/2144e0091fff456a8ecdef1199a22507
I20260928 05:17:59.571626 8500 ts_tablet_manager.cc:1939] T 2144e0091fff456a8ecdef1199a22507 P c35d4d4f8f3a49a09bfaec1f37147dd2: Deleting consensus metadata
I20260928 05:17:59.572002 8477 tablet_bootstrap.cc:492] T f07bad35e54046afb27621bb02449472 P 0f89f414d76944cfb5a9505edaa66d35: Bootstrap starting.
I20260928 05:17:59.570279 8501 raft_consensus.cc:2258] T 2144e0091fff456a8ecdef1199a22507 P eeb06b1625b4449da5c75ca573e0f3e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:17:59.572945 8478 tablet_bootstrap.cc:492] T f07bad35e54046afb27621bb02449472 P c35d4d4f8f3a49a09bfaec1f37147dd2: Bootstrap starting.
I20260928 05:17:59.573210 7953 catalog_manager.cc:4841] TS c35d4d4f8f3a49a09bfaec1f37147dd2 (127.0.97.3:35975): tablet 2144e0091fff456a8ecdef1199a22507 (table test-table [id=e97c41786c094560a03e56916e9fab88]) successfully deleted
I20260928 05:17:59.573544 8477 tablet_bootstrap.cc:654] T f07bad35e54046afb27621bb02449472 P 0f89f414d76944cfb5a9505edaa66d35: Neither blocks nor log segments found. Creating new log.
I20260928 05:17:59.575628 8478 tablet_bootstrap.cc:654] T f07bad35e54046afb27621bb02449472 P c35d4d4f8f3a49a09bfaec1f37147dd2: Neither blocks nor log segments found. Creating new log.
I20260928 05:17:59.575600 8501 ts_tablet_manager.cc:1905] T 2144e0091fff456a8ecdef1199a22507 P eeb06b1625b4449da5c75ca573e0f3e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:17:59.585144 8478 tablet_bootstrap.cc:492] T f07bad35e54046afb27621bb02449472 P c35d4d4f8f3a49a09bfaec1f37147dd2: No bootstrap required, opened a new log
I20260928 05:17:59.585300 8478 ts_tablet_manager.cc:1397] T f07bad35e54046afb27621bb02449472 P c35d4d4f8f3a49a09bfaec1f37147dd2: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.006s
I20260928 05:17:59.587126 8478 raft_consensus.cc:348] T f07bad35e54046afb27621bb02449472 P c35d4d4f8f3a49a09bfaec1f37147dd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f89f414d76944cfb5a9505edaa66d35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42423 } } peers { permanent_uuid: "eeb06b1625b4449da5c75ca573e0f3e6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44359 } } peers { permanent_uuid: "c35d4d4f8f3a49a09bfaec1f37147dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35975 } }
I20260928 05:17:59.587357 8478 raft_consensus.cc:374] T f07bad35e54046afb27621bb02449472 P c35d4d4f8f3a49a09bfaec1f37147dd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:17:59.587435 8478 raft_consensus.cc:729] T f07bad35e54046afb27621bb02449472 P c35d4d4f8f3a49a09bfaec1f37147dd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c35d4d4f8f3a49a09bfaec1f37147dd2, State: Initialized, Role: FOLLOWER
I20260928 05:17:59.587471 8476 tablet_bootstrap.cc:492] T f07bad35e54046afb27621bb02449472 P eeb06b1625b4449da5c75ca573e0f3e6: Bootstrap starting.
I20260928 05:17:59.587685 8478 consensus_queue.cc:260] T f07bad35e54046afb27621bb02449472 P c35d4d4f8f3a49a09bfaec1f37147dd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f89f414d76944cfb5a9505edaa66d35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42423 } } peers { permanent_uuid: "eeb06b1625b4449da5c75ca573e0f3e6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44359 } } peers { permanent_uuid: "c35d4d4f8f3a49a09bfaec1f37147dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35975 } }
I20260928 05:17:59.589248 8476 tablet_bootstrap.cc:654] T f07bad35e54046afb27621bb02449472 P eeb06b1625b4449da5c75ca573e0f3e6: Neither blocks nor log segments found. Creating new log.
I20260928 05:17:59.590559 8478 ts_tablet_manager.cc:1428] T f07bad35e54046afb27621bb02449472 P c35d4d4f8f3a49a09bfaec1f37147dd2: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260928 05:17:59.595243 8477 tablet_bootstrap.cc:492] T f07bad35e54046afb27621bb02449472 P 0f89f414d76944cfb5a9505edaa66d35: No bootstrap required, opened a new log
I20260928 05:17:59.595391 8477 ts_tablet_manager.cc:1397] T f07bad35e54046afb27621bb02449472 P 0f89f414d76944cfb5a9505edaa66d35: Time spent bootstrapping tablet: real 0.023s user 0.010s sys 0.013s
I20260928 05:17:59.595729 8501 ts_tablet_manager.cc:1918] T 2144e0091fff456a8ecdef1199a22507 P eeb06b1625b4449da5c75ca573e0f3e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:17:59.595950 8501 log.cc:1196] T 2144e0091fff456a8ecdef1199a22507 P eeb06b1625b4449da5c75ca573e0f3e6: Deleting WAL directory at /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/minicluster-data/ts-1/wal/wals/2144e0091fff456a8ecdef1199a22507
I20260928 05:17:59.596086 8477 raft_consensus.cc:348] T f07bad35e54046afb27621bb02449472 P 0f89f414d76944cfb5a9505edaa66d35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f89f414d76944cfb5a9505edaa66d35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42423 } } peers { permanent_uuid: "eeb06b1625b4449da5c75ca573e0f3e6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44359 } } peers { permanent_uuid: "c35d4d4f8f3a49a09bfaec1f37147dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35975 } }
I20260928 05:17:59.596288 8477 raft_consensus.cc:374] T f07bad35e54046afb27621bb02449472 P 0f89f414d76944cfb5a9505edaa66d35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:17:59.596376 8477 raft_consensus.cc:729] T f07bad35e54046afb27621bb02449472 P 0f89f414d76944cfb5a9505edaa66d35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f89f414d76944cfb5a9505edaa66d35, State: Initialized, Role: FOLLOWER
I20260928 05:17:59.596684 8501 ts_tablet_manager.cc:1939] T 2144e0091fff456a8ecdef1199a22507 P eeb06b1625b4449da5c75ca573e0f3e6: Deleting consensus metadata
I20260928 05:17:59.596655 8477 consensus_queue.cc:260] T f07bad35e54046afb27621bb02449472 P 0f89f414d76944cfb5a9505edaa66d35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f89f414d76944cfb5a9505edaa66d35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42423 } } peers { permanent_uuid: "eeb06b1625b4449da5c75ca573e0f3e6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44359 } } peers { permanent_uuid: "c35d4d4f8f3a49a09bfaec1f37147dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35975 } }
I20260928 05:17:59.598781 7951 catalog_manager.cc:4841] TS eeb06b1625b4449da5c75ca573e0f3e6 (127.0.97.2:44359): tablet 2144e0091fff456a8ecdef1199a22507 (table test-table [id=e97c41786c094560a03e56916e9fab88]) successfully deleted
I20260928 05:17:59.599644 8477 ts_tablet_manager.cc:1428] T f07bad35e54046afb27621bb02449472 P 0f89f414d76944cfb5a9505edaa66d35: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260928 05:17:59.603101 8476 tablet_bootstrap.cc:492] T f07bad35e54046afb27621bb02449472 P eeb06b1625b4449da5c75ca573e0f3e6: No bootstrap required, opened a new log
I20260928 05:17:59.603235 8476 ts_tablet_manager.cc:1397] T f07bad35e54046afb27621bb02449472 P eeb06b1625b4449da5c75ca573e0f3e6: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.008s
I20260928 05:17:59.603811 8476 raft_consensus.cc:348] T f07bad35e54046afb27621bb02449472 P eeb06b1625b4449da5c75ca573e0f3e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f89f414d76944cfb5a9505edaa66d35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42423 } } peers { permanent_uuid: "eeb06b1625b4449da5c75ca573e0f3e6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44359 } } peers { permanent_uuid: "c35d4d4f8f3a49a09bfaec1f37147dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35975 } }
I20260928 05:17:59.603971 8476 raft_consensus.cc:374] T f07bad35e54046afb27621bb02449472 P eeb06b1625b4449da5c75ca573e0f3e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:17:59.604069 8476 raft_consensus.cc:729] T f07bad35e54046afb27621bb02449472 P eeb06b1625b4449da5c75ca573e0f3e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeb06b1625b4449da5c75ca573e0f3e6, State: Initialized, Role: FOLLOWER
I20260928 05:17:59.604300 8476 consensus_queue.cc:260] T f07bad35e54046afb27621bb02449472 P eeb06b1625b4449da5c75ca573e0f3e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f89f414d76944cfb5a9505edaa66d35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42423 } } peers { permanent_uuid: "eeb06b1625b4449da5c75ca573e0f3e6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44359 } } peers { permanent_uuid: "c35d4d4f8f3a49a09bfaec1f37147dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35975 } }
I20260928 05:17:59.604938 8476 ts_tablet_manager.cc:1428] T f07bad35e54046afb27621bb02449472 P eeb06b1625b4449da5c75ca573e0f3e6: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 05:17:59.695674 8484 raft_consensus.cc:482] T f07bad35e54046afb27621bb02449472 P c35d4d4f8f3a49a09bfaec1f37147dd2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:17:59.695911 8484 raft_consensus.cc:504] T f07bad35e54046afb27621bb02449472 P c35d4d4f8f3a49a09bfaec1f37147dd2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f89f414d76944cfb5a9505edaa66d35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42423 } } peers { permanent_uuid: "eeb06b1625b4449da5c75ca573e0f3e6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44359 } } peers { permanent_uuid: "c35d4d4f8f3a49a09bfaec1f37147dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35975 } }
I20260928 05:17:59.697562 8484 leader_election.cc:288] T f07bad35e54046afb27621bb02449472 P c35d4d4f8f3a49a09bfaec1f37147dd2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f89f414d76944cfb5a9505edaa66d35 (127.0.97.1:42423), eeb06b1625b4449da5c75ca573e0f3e6 (127.0.97.2:44359)
I20260928 05:17:59.704762 8472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:59.698037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41247 (local address 127.0.97.1:42423)
0928 05:17:59.698219 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:59.698226 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:59.698310 (+ 84us) server_negotiation.cc:407] Connection header received
0928 05:17:59.698601 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:59.698606 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:59.698692 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:59.698754 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:17:59.699464 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:59.700568 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:59.703107 (+ 2539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:59.703887 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:59.703996 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:59.704092 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:59.704358 (+ 266us) server_negotiation.cc:299] Negotiation successful
0928 05:17:59.704478 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 05:17:59.705374 8506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:59.697736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42423 (local address 127.0.97.3:41247)
0928 05:17:59.698223 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:59.698241 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:59.698422 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:59.698937 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:59.698941 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:59.698962 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:17:59.699266 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:59.699276 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:59.700701 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:59.700706 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:59.702908 (+ 2202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:59.702921 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:59.704063 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:59.704068 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:59.704211 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:59.704235 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 05:17:59.704314 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 05:17:59.704446 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"mutex_wait_us":57,"thread_start_us":491,"threads_started":1}
I20260928 05:17:59.705555 8471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:59.698207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53699 (local address 127.0.97.2:44359)
0928 05:17:59.698373 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:59.698379 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:59.698889 (+ 510us) server_negotiation.cc:407] Connection header received
0928 05:17:59.699017 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:59.699023 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:59.699093 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:59.699185 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:17:59.699739 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:59.700484 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:59.703082 (+ 2598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:59.704333 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:59.704484 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:59.704573 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:59.705138 (+ 565us) server_negotiation.cc:299] Negotiation successful
0928 05:17:59.705273 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 05:17:59.705611 8507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:59.697832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44359 (local address 127.0.97.3:53699)
0928 05:17:59.698791 (+ 959us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:59.698806 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:59.698914 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:59.699238 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:59.699243 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:59.699269 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:17:59.699601 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:59.699609 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:59.700619 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:59.700623 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:59.702911 (+ 2288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:59.702921 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:59.704746 (+ 1825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:59.704753 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:59.704963 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:59.704997 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 05:17:59.705091 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 05:17:59.705218 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":852,"mutex_wait_us":29,"thread_start_us":651,"threads_started":1}
I20260928 05:17:59.706343 8124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f07bad35e54046afb27621bb02449472" candidate_uuid: "c35d4d4f8f3a49a09bfaec1f37147dd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f89f414d76944cfb5a9505edaa66d35" is_pre_election: true
I20260928 05:17:59.706570 8124 raft_consensus.cc:2454] T f07bad35e54046afb27621bb02449472 P 0f89f414d76944cfb5a9505edaa66d35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c35d4d4f8f3a49a09bfaec1f37147dd2 in term 0.
I20260928 05:17:59.706480 8268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f07bad35e54046afb27621bb02449472" candidate_uuid: "c35d4d4f8f3a49a09bfaec1f37147dd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeb06b1625b4449da5c75ca573e0f3e6" is_pre_election: true
I20260928 05:17:59.706895 8268 raft_consensus.cc:2454] T f07bad35e54046afb27621bb02449472 P eeb06b1625b4449da5c75ca573e0f3e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c35d4d4f8f3a49a09bfaec1f37147dd2 in term 0.
I20260928 05:17:59.707017 8336 leader_election.cc:302] T f07bad35e54046afb27621bb02449472 P c35d4d4f8f3a49a09bfaec1f37147dd2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f89f414d76944cfb5a9505edaa66d35, c35d4d4f8f3a49a09bfaec1f37147dd2; no voters:
I20260928 05:17:59.707388 8484 raft_consensus.cc:2781] T f07bad35e54046afb27621bb02449472 P c35d4d4f8f3a49a09bfaec1f37147dd2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 05:17:59.707500 8484 raft_consensus.cc:482] T f07bad35e54046afb27621bb02449472 P c35d4d4f8f3a49a09bfaec1f37147dd2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 05:17:59.707577 8484 raft_consensus.cc:3037] T f07bad35e54046afb27621bb02449472 P c35d4d4f8f3a49a09bfaec1f37147dd2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:17:59.709594 8484 raft_consensus.cc:504] T f07bad35e54046afb27621bb02449472 P c35d4d4f8f3a49a09bfaec1f37147dd2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f89f414d76944cfb5a9505edaa66d35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42423 } } peers { permanent_uuid: "eeb06b1625b4449da5c75ca573e0f3e6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44359 } } peers { permanent_uuid: "c35d4d4f8f3a49a09bfaec1f37147dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35975 } }
I20260928 05:17:59.710119 8484 leader_election.cc:288] T f07bad35e54046afb27621bb02449472 P c35d4d4f8f3a49a09bfaec1f37147dd2 [CANDIDATE]: Term 1 election: Requested vote from peers 0f89f414d76944cfb5a9505edaa66d35 (127.0.97.1:42423), eeb06b1625b4449da5c75ca573e0f3e6 (127.0.97.2:44359)
I20260928 05:17:59.710479 8124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f07bad35e54046afb27621bb02449472" candidate_uuid: "c35d4d4f8f3a49a09bfaec1f37147dd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f89f414d76944cfb5a9505edaa66d35"
I20260928 05:17:59.710580 8268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f07bad35e54046afb27621bb02449472" candidate_uuid: "c35d4d4f8f3a49a09bfaec1f37147dd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeb06b1625b4449da5c75ca573e0f3e6"
I20260928 05:17:59.710681 8124 raft_consensus.cc:3037] T f07bad35e54046afb27621bb02449472 P 0f89f414d76944cfb5a9505edaa66d35 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:17:59.710775 8268 raft_consensus.cc:3037] T f07bad35e54046afb27621bb02449472 P eeb06b1625b4449da5c75ca573e0f3e6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:17:59.713500 8124 raft_consensus.cc:2454] T f07bad35e54046afb27621bb02449472 P 0f89f414d76944cfb5a9505edaa66d35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c35d4d4f8f3a49a09bfaec1f37147dd2 in term 1.
I20260928 05:17:59.713790 8268 raft_consensus.cc:2454] T f07bad35e54046afb27621bb02449472 P eeb06b1625b4449da5c75ca573e0f3e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c35d4d4f8f3a49a09bfaec1f37147dd2 in term 1.
I20260928 05:17:59.713913 8336 leader_election.cc:302] T f07bad35e54046afb27621bb02449472 P c35d4d4f8f3a49a09bfaec1f37147dd2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f89f414d76944cfb5a9505edaa66d35, c35d4d4f8f3a49a09bfaec1f37147dd2; no voters:
I20260928 05:17:59.714190 8484 raft_consensus.cc:2781] T f07bad35e54046afb27621bb02449472 P c35d4d4f8f3a49a09bfaec1f37147dd2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:17:59.714645 8484 raft_consensus.cc:686] T f07bad35e54046afb27621bb02449472 P c35d4d4f8f3a49a09bfaec1f37147dd2 [term 1 LEADER]: Becoming Leader. State: Replica: c35d4d4f8f3a49a09bfaec1f37147dd2, State: Running, Role: LEADER
I20260928 05:17:59.715031 8484 consensus_queue.cc:237] T f07bad35e54046afb27621bb02449472 P c35d4d4f8f3a49a09bfaec1f37147dd2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f89f414d76944cfb5a9505edaa66d35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42423 } } peers { permanent_uuid: "eeb06b1625b4449da5c75ca573e0f3e6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44359 } } peers { permanent_uuid: "c35d4d4f8f3a49a09bfaec1f37147dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35975 } }
I20260928 05:17:59.718540 7966 catalog_manager.cc:5495] T f07bad35e54046afb27621bb02449472 P c35d4d4f8f3a49a09bfaec1f37147dd2 reported cstate change: term changed from 0 to 1, leader changed from <none> to c35d4d4f8f3a49a09bfaec1f37147dd2 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "c35d4d4f8f3a49a09bfaec1f37147dd2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f89f414d76944cfb5a9505edaa66d35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eeb06b1625b4449da5c75ca573e0f3e6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c35d4d4f8f3a49a09bfaec1f37147dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35975 } health_report { overall_health: HEALTHY } } }
Sep 28 05:17:59 dist-test-slave-whsn krb5kdc[7922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572677, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260928 05:17:59.787621 8515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:59.771404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39261 (local address 127.0.0.1:53244)
0928 05:17:59.773370 (+ 1966us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:59.773391 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:59.773545 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:59.774358 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:59.774363 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:59.774974 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:17:59.775187 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:59.775196 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:59.776711 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:59.776716 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:59.777772 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:59.777785 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:59.778704 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:59.778709 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:59.778853 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:59.779499 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:17:59.779518 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:17:59.781676 (+ 2158us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:17:59.785139 (+ 3463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:59.785148 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:59.785156 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:59.785695 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:59.786071 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:17:59.786075 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:17:59.786077 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:17:59.786213 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:17:59.786778 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:17:59.786783 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:17:59.786938 (+ 155us) client_negotiation.cc:769] Sending connection context
0928 05:17:59.787077 (+ 139us) client_negotiation.cc:240] Negotiation successful
0928 05:17:59.787261 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1805,"thread_start_us":1644,"threads_started":1}
I20260928 05:17:59.787873 8516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:59.771415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53244 (local address 127.0.97.62:39261)
0928 05:17:59.773984 (+ 2569us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:59.773989 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:59.774012 (+ 23us) server_negotiation.cc:407] Connection header received
0928 05:17:59.774063 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:59.774067 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:59.774135 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:59.774240 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:17:59.775351 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:59.776527 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:59.777921 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:59.778538 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:59.778657 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:59.781869 (+ 3212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:17:59.781875 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:17:59.781880 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:17:59.781915 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:17:59.785016 (+ 3101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:59.785815 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:59.785821 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:59.785825 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:59.785963 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:17:59.786320 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:17:59.786326 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:17:59.786329 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:17:59.786664 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:17:59.786798 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:59.787205 (+ 407us) server_negotiation.cc:299] Negotiation successful
0928 05:17:59.787405 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2456,"spinlock_wait_cycles":12160,"thread_start_us":2322,"threads_started":1}
I20260928 05:17:59.792898 7966 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53244:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 05:17:59.793951 7966 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 05:17:59.800798 8242 tablet_service.cc:1467] Processing CreateTablet for tablet ea5bb4026f664d0fab67f0e7e7c70f5e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=52b4cc5df482425cbff49ef825797079]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 05:17:59.801069 8381 tablet_service.cc:1467] Processing CreateTablet for tablet ea5bb4026f664d0fab67f0e7e7c70f5e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=52b4cc5df482425cbff49ef825797079]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 05:17:59.801051 8104 tablet_service.cc:1467] Processing CreateTablet for tablet ea5bb4026f664d0fab67f0e7e7c70f5e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=52b4cc5df482425cbff49ef825797079]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 05:17:59.801385 8242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea5bb4026f664d0fab67f0e7e7c70f5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:17:59.801553 8381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea5bb4026f664d0fab67f0e7e7c70f5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:17:59.801568 8104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea5bb4026f664d0fab67f0e7e7c70f5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:17:59.808400 8476 tablet_bootstrap.cc:492] T ea5bb4026f664d0fab67f0e7e7c70f5e P eeb06b1625b4449da5c75ca573e0f3e6: Bootstrap starting.
I20260928 05:17:59.810177 8476 tablet_bootstrap.cc:654] T ea5bb4026f664d0fab67f0e7e7c70f5e P eeb06b1625b4449da5c75ca573e0f3e6: Neither blocks nor log segments found. Creating new log.
I20260928 05:17:59.811136 8478 tablet_bootstrap.cc:492] T ea5bb4026f664d0fab67f0e7e7c70f5e P c35d4d4f8f3a49a09bfaec1f37147dd2: Bootstrap starting.
I20260928 05:17:59.812717 8478 tablet_bootstrap.cc:654] T ea5bb4026f664d0fab67f0e7e7c70f5e P c35d4d4f8f3a49a09bfaec1f37147dd2: Neither blocks nor log segments found. Creating new log.
I20260928 05:17:59.813880 8477 tablet_bootstrap.cc:492] T ea5bb4026f664d0fab67f0e7e7c70f5e P 0f89f414d76944cfb5a9505edaa66d35: Bootstrap starting.
I20260928 05:17:59.815349 8477 tablet_bootstrap.cc:654] T ea5bb4026f664d0fab67f0e7e7c70f5e P 0f89f414d76944cfb5a9505edaa66d35: Neither blocks nor log segments found. Creating new log.
I20260928 05:17:59.816617 8478 tablet_bootstrap.cc:492] T ea5bb4026f664d0fab67f0e7e7c70f5e P c35d4d4f8f3a49a09bfaec1f37147dd2: No bootstrap required, opened a new log
I20260928 05:17:59.816754 8478 ts_tablet_manager.cc:1397] T ea5bb4026f664d0fab67f0e7e7c70f5e P c35d4d4f8f3a49a09bfaec1f37147dd2: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260928 05:17:59.817370 8478 raft_consensus.cc:348] T ea5bb4026f664d0fab67f0e7e7c70f5e P c35d4d4f8f3a49a09bfaec1f37147dd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f89f414d76944cfb5a9505edaa66d35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42423 } } peers { permanent_uuid: "eeb06b1625b4449da5c75ca573e0f3e6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44359 } } peers { permanent_uuid: "c35d4d4f8f3a49a09bfaec1f37147dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35975 } }
I20260928 05:17:59.817571 8478 raft_consensus.cc:374] T ea5bb4026f664d0fab67f0e7e7c70f5e P c35d4d4f8f3a49a09bfaec1f37147dd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:17:59.817649 8478 raft_consensus.cc:729] T ea5bb4026f664d0fab67f0e7e7c70f5e P c35d4d4f8f3a49a09bfaec1f37147dd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c35d4d4f8f3a49a09bfaec1f37147dd2, State: Initialized, Role: FOLLOWER
I20260928 05:17:59.817874 8478 consensus_queue.cc:260] T ea5bb4026f664d0fab67f0e7e7c70f5e P c35d4d4f8f3a49a09bfaec1f37147dd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f89f414d76944cfb5a9505edaa66d35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42423 } } peers { permanent_uuid: "eeb06b1625b4449da5c75ca573e0f3e6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44359 } } peers { permanent_uuid: "c35d4d4f8f3a49a09bfaec1f37147dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35975 } }
I20260928 05:17:59.818265 8484 tablet_replica.cc:439] T ea5bb4026f664d0fab67f0e7e7c70f5e P c35d4d4f8f3a49a09bfaec1f37147dd2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f89f414d76944cfb5a9505edaa66d35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42423 } } peers { permanent_uuid: "eeb06b1625b4449da5c75ca573e0f3e6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44359 } } peers { permanent_uuid: "c35d4d4f8f3a49a09bfaec1f37147dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35975 } } }
I20260928 05:17:59.818435 8478 ts_tablet_manager.cc:1428] T ea5bb4026f664d0fab67f0e7e7c70f5e P c35d4d4f8f3a49a09bfaec1f37147dd2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260928 05:17:59.818571 8484 tablet_replica.cc:442] T ea5bb4026f664d0fab67f0e7e7c70f5e P c35d4d4f8f3a49a09bfaec1f37147dd2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 05:17:59.820348 8477 tablet_bootstrap.cc:492] T ea5bb4026f664d0fab67f0e7e7c70f5e P 0f89f414d76944cfb5a9505edaa66d35: No bootstrap required, opened a new log
I20260928 05:17:59.820406 8476 tablet_bootstrap.cc:492] T ea5bb4026f664d0fab67f0e7e7c70f5e P eeb06b1625b4449da5c75ca573e0f3e6: No bootstrap required, opened a new log
I20260928 05:17:59.820505 8477 ts_tablet_manager.cc:1397] T ea5bb4026f664d0fab67f0e7e7c70f5e P 0f89f414d76944cfb5a9505edaa66d35: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260928 05:17:59.820559 8476 ts_tablet_manager.cc:1397] T ea5bb4026f664d0fab67f0e7e7c70f5e P eeb06b1625b4449da5c75ca573e0f3e6: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20260928 05:17:59.821261 8477 raft_consensus.cc:348] T ea5bb4026f664d0fab67f0e7e7c70f5e P 0f89f414d76944cfb5a9505edaa66d35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f89f414d76944cfb5a9505edaa66d35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42423 } } peers { permanent_uuid: "eeb06b1625b4449da5c75ca573e0f3e6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44359 } } peers { permanent_uuid: "c35d4d4f8f3a49a09bfaec1f37147dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35975 } }
I20260928 05:17:59.821326 8476 raft_consensus.cc:348] T ea5bb4026f664d0fab67f0e7e7c70f5e P eeb06b1625b4449da5c75ca573e0f3e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f89f414d76944cfb5a9505edaa66d35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42423 } } peers { permanent_uuid: "eeb06b1625b4449da5c75ca573e0f3e6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44359 } } peers { permanent_uuid: "c35d4d4f8f3a49a09bfaec1f37147dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35975 } }
I20260928 05:17:59.821504 8477 raft_consensus.cc:374] T ea5bb4026f664d0fab67f0e7e7c70f5e P 0f89f414d76944cfb5a9505edaa66d35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:17:59.821583 8476 raft_consensus.cc:374] T ea5bb4026f664d0fab67f0e7e7c70f5e P eeb06b1625b4449da5c75ca573e0f3e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:17:59.821590 8477 raft_consensus.cc:729] T ea5bb4026f664d0fab67f0e7e7c70f5e P 0f89f414d76944cfb5a9505edaa66d35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f89f414d76944cfb5a9505edaa66d35, State: Initialized, Role: FOLLOWER
I20260928 05:17:59.821672 8476 raft_consensus.cc:729] T ea5bb4026f664d0fab67f0e7e7c70f5e P eeb06b1625b4449da5c75ca573e0f3e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeb06b1625b4449da5c75ca573e0f3e6, State: Initialized, Role: FOLLOWER
I20260928 05:17:59.821823 8477 consensus_queue.cc:260] T ea5bb4026f664d0fab67f0e7e7c70f5e P 0f89f414d76944cfb5a9505edaa66d35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f89f414d76944cfb5a9505edaa66d35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42423 } } peers { permanent_uuid: "eeb06b1625b4449da5c75ca573e0f3e6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44359 } } peers { permanent_uuid: "c35d4d4f8f3a49a09bfaec1f37147dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35975 } }
I20260928 05:17:59.821981 8476 consensus_queue.cc:260] T ea5bb4026f664d0fab67f0e7e7c70f5e P eeb06b1625b4449da5c75ca573e0f3e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f89f414d76944cfb5a9505edaa66d35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42423 } } peers { permanent_uuid: "eeb06b1625b4449da5c75ca573e0f3e6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44359 } } peers { permanent_uuid: "c35d4d4f8f3a49a09bfaec1f37147dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35975 } }
I20260928 05:17:59.822307 8482 tablet_replica.cc:439] T ea5bb4026f664d0fab67f0e7e7c70f5e P 0f89f414d76944cfb5a9505edaa66d35: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f89f414d76944cfb5a9505edaa66d35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42423 } } peers { permanent_uuid: "eeb06b1625b4449da5c75ca573e0f3e6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44359 } } peers { permanent_uuid: "c35d4d4f8f3a49a09bfaec1f37147dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35975 } } }
I20260928 05:17:59.822432 8483 tablet_replica.cc:439] T ea5bb4026f664d0fab67f0e7e7c70f5e P eeb06b1625b4449da5c75ca573e0f3e6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f89f414d76944cfb5a9505edaa66d35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42423 } } peers { permanent_uuid: "eeb06b1625b4449da5c75ca573e0f3e6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44359 } } peers { permanent_uuid: "c35d4d4f8f3a49a09bfaec1f37147dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35975 } } }
I20260928 05:17:59.822674 8482 tablet_replica.cc:442] T ea5bb4026f664d0fab67f0e7e7c70f5e P 0f89f414d76944cfb5a9505edaa66d35: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 05:17:59.822676 8483 tablet_replica.cc:442] T ea5bb4026f664d0fab67f0e7e7c70f5e P eeb06b1625b4449da5c75ca573e0f3e6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 05:17:59.823120 8477 ts_tablet_manager.cc:1428] T ea5bb4026f664d0fab67f0e7e7c70f5e P 0f89f414d76944cfb5a9505edaa66d35: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260928 05:17:59.823525 8476 ts_tablet_manager.cc:1428] T ea5bb4026f664d0fab67f0e7e7c70f5e P eeb06b1625b4449da5c75ca573e0f3e6: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260928 05:17:59.856288 8484 raft_consensus.cc:482] T ea5bb4026f664d0fab67f0e7e7c70f5e P c35d4d4f8f3a49a09bfaec1f37147dd2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:17:59.856463 8484 raft_consensus.cc:504] T ea5bb4026f664d0fab67f0e7e7c70f5e P c35d4d4f8f3a49a09bfaec1f37147dd2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f89f414d76944cfb5a9505edaa66d35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42423 } } peers { permanent_uuid: "eeb06b1625b4449da5c75ca573e0f3e6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44359 } } peers { permanent_uuid: "c35d4d4f8f3a49a09bfaec1f37147dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35975 } }
I20260928 05:17:59.857070 8484 leader_election.cc:288] T ea5bb4026f664d0fab67f0e7e7c70f5e P c35d4d4f8f3a49a09bfaec1f37147dd2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f89f414d76944cfb5a9505edaa66d35 (127.0.97.1:42423), eeb06b1625b4449da5c75ca573e0f3e6 (127.0.97.2:44359)
I20260928 05:17:59.857574 8268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea5bb4026f664d0fab67f0e7e7c70f5e" candidate_uuid: "c35d4d4f8f3a49a09bfaec1f37147dd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeb06b1625b4449da5c75ca573e0f3e6" is_pre_election: true
I20260928 05:17:59.857599 8124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea5bb4026f664d0fab67f0e7e7c70f5e" candidate_uuid: "c35d4d4f8f3a49a09bfaec1f37147dd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f89f414d76944cfb5a9505edaa66d35" is_pre_election: true
I20260928 05:17:59.857829 8124 raft_consensus.cc:2454] T ea5bb4026f664d0fab67f0e7e7c70f5e P 0f89f414d76944cfb5a9505edaa66d35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c35d4d4f8f3a49a09bfaec1f37147dd2 in term 0.
I20260928 05:17:59.857839 8268 raft_consensus.cc:2454] T ea5bb4026f664d0fab67f0e7e7c70f5e P eeb06b1625b4449da5c75ca573e0f3e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c35d4d4f8f3a49a09bfaec1f37147dd2 in term 0.
I20260928 05:17:59.858207 8333 leader_election.cc:302] T ea5bb4026f664d0fab67f0e7e7c70f5e P c35d4d4f8f3a49a09bfaec1f37147dd2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c35d4d4f8f3a49a09bfaec1f37147dd2, eeb06b1625b4449da5c75ca573e0f3e6; no voters:
I20260928 05:17:59.858525 8484 raft_consensus.cc:2781] T ea5bb4026f664d0fab67f0e7e7c70f5e P c35d4d4f8f3a49a09bfaec1f37147dd2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 05:17:59.858635 8484 raft_consensus.cc:482] T ea5bb4026f664d0fab67f0e7e7c70f5e P c35d4d4f8f3a49a09bfaec1f37147dd2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 05:17:59.858744 8484 raft_consensus.cc:3037] T ea5bb4026f664d0fab67f0e7e7c70f5e P c35d4d4f8f3a49a09bfaec1f37147dd2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:17:59.861688 8484 raft_consensus.cc:504] T ea5bb4026f664d0fab67f0e7e7c70f5e P c35d4d4f8f3a49a09bfaec1f37147dd2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f89f414d76944cfb5a9505edaa66d35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42423 } } peers { permanent_uuid: "eeb06b1625b4449da5c75ca573e0f3e6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44359 } } peers { permanent_uuid: "c35d4d4f8f3a49a09bfaec1f37147dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35975 } }
I20260928 05:17:59.862614 8124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea5bb4026f664d0fab67f0e7e7c70f5e" candidate_uuid: "c35d4d4f8f3a49a09bfaec1f37147dd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f89f414d76944cfb5a9505edaa66d35"
I20260928 05:17:59.862746 8268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea5bb4026f664d0fab67f0e7e7c70f5e" candidate_uuid: "c35d4d4f8f3a49a09bfaec1f37147dd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeb06b1625b4449da5c75ca573e0f3e6"
I20260928 05:17:59.862824 8124 raft_consensus.cc:3037] T ea5bb4026f664d0fab67f0e7e7c70f5e P 0f89f414d76944cfb5a9505edaa66d35 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:17:59.862979 8268 raft_consensus.cc:3037] T ea5bb4026f664d0fab67f0e7e7c70f5e P eeb06b1625b4449da5c75ca573e0f3e6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:17:59.863185 8484 leader_election.cc:288] T ea5bb4026f664d0fab67f0e7e7c70f5e P c35d4d4f8f3a49a09bfaec1f37147dd2 [CANDIDATE]: Term 1 election: Requested vote from peers 0f89f414d76944cfb5a9505edaa66d35 (127.0.97.1:42423), eeb06b1625b4449da5c75ca573e0f3e6 (127.0.97.2:44359)
I20260928 05:17:59.864775 8124 raft_consensus.cc:2454] T ea5bb4026f664d0fab67f0e7e7c70f5e P 0f89f414d76944cfb5a9505edaa66d35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c35d4d4f8f3a49a09bfaec1f37147dd2 in term 1.
I20260928 05:17:59.865160 8336 leader_election.cc:302] T ea5bb4026f664d0fab67f0e7e7c70f5e P c35d4d4f8f3a49a09bfaec1f37147dd2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f89f414d76944cfb5a9505edaa66d35, c35d4d4f8f3a49a09bfaec1f37147dd2; no voters:
I20260928 05:17:59.865384 8484 raft_consensus.cc:2781] T ea5bb4026f664d0fab67f0e7e7c70f5e P c35d4d4f8f3a49a09bfaec1f37147dd2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:17:59.865549 8484 raft_consensus.cc:686] T ea5bb4026f664d0fab67f0e7e7c70f5e P c35d4d4f8f3a49a09bfaec1f37147dd2 [term 1 LEADER]: Becoming Leader. State: Replica: c35d4d4f8f3a49a09bfaec1f37147dd2, State: Running, Role: LEADER
I20260928 05:17:59.865760 8484 consensus_queue.cc:237] T ea5bb4026f664d0fab67f0e7e7c70f5e P c35d4d4f8f3a49a09bfaec1f37147dd2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f89f414d76944cfb5a9505edaa66d35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42423 } } peers { permanent_uuid: "eeb06b1625b4449da5c75ca573e0f3e6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44359 } } peers { permanent_uuid: "c35d4d4f8f3a49a09bfaec1f37147dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35975 } }
I20260928 05:17:59.865983 8268 raft_consensus.cc:2454] T ea5bb4026f664d0fab67f0e7e7c70f5e P eeb06b1625b4449da5c75ca573e0f3e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c35d4d4f8f3a49a09bfaec1f37147dd2 in term 1.
I20260928 05:17:59.867300 8508 tablet_replica.cc:439] T ea5bb4026f664d0fab67f0e7e7c70f5e P c35d4d4f8f3a49a09bfaec1f37147dd2: TxnStatusTablet state changed. Reason: New leader c35d4d4f8f3a49a09bfaec1f37147dd2. Latest consensus state: current_term: 1 leader_uuid: "c35d4d4f8f3a49a09bfaec1f37147dd2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f89f414d76944cfb5a9505edaa66d35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42423 } } peers { permanent_uuid: "eeb06b1625b4449da5c75ca573e0f3e6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44359 } } peers { permanent_uuid: "c35d4d4f8f3a49a09bfaec1f37147dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35975 } } }
I20260928 05:17:59.867702 8508 tablet_replica.cc:442] T ea5bb4026f664d0fab67f0e7e7c70f5e P c35d4d4f8f3a49a09bfaec1f37147dd2: This TxnStatusTablet replica's current role is: LEADER
I20260928 05:17:59.868400 8521 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 05:17:59.868299 7967 catalog_manager.cc:5495] T ea5bb4026f664d0fab67f0e7e7c70f5e P c35d4d4f8f3a49a09bfaec1f37147dd2 reported cstate change: term changed from 0 to 1, leader changed from <none> to c35d4d4f8f3a49a09bfaec1f37147dd2 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "c35d4d4f8f3a49a09bfaec1f37147dd2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f89f414d76944cfb5a9505edaa66d35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eeb06b1625b4449da5c75ca573e0f3e6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c35d4d4f8f3a49a09bfaec1f37147dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35975 } health_report { overall_health: HEALTHY } } }
I20260928 05:17:59.868607 8521 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 05:17:59.886994 8473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:59.882168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43414 (local address 127.0.97.3:35975)
0928 05:17:59.882326 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:59.882331 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:59.882346 (+ 15us) server_negotiation.cc:407] Connection header received
0928 05:17:59.882434 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:59.882437 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:59.882502 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:59.882619 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:17:59.882959 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:59.883724 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:59.885000 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:59.885422 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:59.885521 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:59.885963 (+ 442us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:17:59.886411 (+ 448us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:17:59.886532 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:59.886621 (+ 89us) server_negotiation.cc:299] Negotiation successful
0928 05:17:59.886694 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 05:17:59.887014 8515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:59.881790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35975 (local address 127.0.0.1:43414)
0928 05:17:59.882003 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:59.882021 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:59.882157 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:59.882663 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:59.882668 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:59.882680 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 05:17:59.882840 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:59.882848 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:59.883839 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:59.883845 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:59.884888 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:59.884903 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:59.885556 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:59.885562 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:59.885758 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:59.885805 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 05:17:59.886513 (+ 708us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 05:17:59.886531 (+ 18us) client_negotiation.cc:769] Sending connection context
0928 05:17:59.886604 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 05:17:59.886739 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260928 05:17:59.890513 8124 raft_consensus.cc:1261] T ea5bb4026f664d0fab67f0e7e7c70f5e P 0f89f414d76944cfb5a9505edaa66d35 [term 1 FOLLOWER]: Refusing update from remote peer c35d4d4f8f3a49a09bfaec1f37147dd2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:17:59.890905 8268 raft_consensus.cc:1261] T ea5bb4026f664d0fab67f0e7e7c70f5e P eeb06b1625b4449da5c75ca573e0f3e6 [term 1 FOLLOWER]: Refusing update from remote peer c35d4d4f8f3a49a09bfaec1f37147dd2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:17:59.891386 8508 consensus_queue.cc:1035] T ea5bb4026f664d0fab67f0e7e7c70f5e P c35d4d4f8f3a49a09bfaec1f37147dd2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f89f414d76944cfb5a9505edaa66d35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:17:59.891983 8508 consensus_queue.cc:1035] T ea5bb4026f664d0fab67f0e7e7c70f5e P c35d4d4f8f3a49a09bfaec1f37147dd2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeb06b1625b4449da5c75ca573e0f3e6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:17:59.895778 8508 tablet_replica.cc:439] T ea5bb4026f664d0fab67f0e7e7c70f5e P c35d4d4f8f3a49a09bfaec1f37147dd2: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c35d4d4f8f3a49a09bfaec1f37147dd2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f89f414d76944cfb5a9505edaa66d35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42423 } } peers { permanent_uuid: "eeb06b1625b4449da5c75ca573e0f3e6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44359 } } peers { permanent_uuid: "c35d4d4f8f3a49a09bfaec1f37147dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35975 } } }
I20260928 05:17:59.895984 8508 tablet_replica.cc:442] T ea5bb4026f664d0fab67f0e7e7c70f5e P c35d4d4f8f3a49a09bfaec1f37147dd2: This TxnStatusTablet replica's current role is: LEADER
I20260928 05:17:59.896929 8492 mvcc.cc:205] Tried to move back new op lower bound from 7334185696824958976 to 7334185696732229632. Current Snapshot: MvccSnapshot[applied={T|T < 7334185696824958976 or (T in {7334185696824958976})}]
I20260928 05:17:59.907341 8482 tablet_replica.cc:439] T ea5bb4026f664d0fab67f0e7e7c70f5e P 0f89f414d76944cfb5a9505edaa66d35: TxnStatusTablet state changed. Reason: New leader c35d4d4f8f3a49a09bfaec1f37147dd2. Latest consensus state: current_term: 1 leader_uuid: "c35d4d4f8f3a49a09bfaec1f37147dd2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f89f414d76944cfb5a9505edaa66d35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42423 } } peers { permanent_uuid: "eeb06b1625b4449da5c75ca573e0f3e6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44359 } } peers { permanent_uuid: "c35d4d4f8f3a49a09bfaec1f37147dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35975 } } }
I20260928 05:17:59.907639 8482 tablet_replica.cc:442] T ea5bb4026f664d0fab67f0e7e7c70f5e P 0f89f414d76944cfb5a9505edaa66d35: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 05:17:59.910905 8524 tablet_replica.cc:439] T ea5bb4026f664d0fab67f0e7e7c70f5e P c35d4d4f8f3a49a09bfaec1f37147dd2: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c35d4d4f8f3a49a09bfaec1f37147dd2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f89f414d76944cfb5a9505edaa66d35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42423 } } peers { permanent_uuid: "eeb06b1625b4449da5c75ca573e0f3e6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44359 } } peers { permanent_uuid: "c35d4d4f8f3a49a09bfaec1f37147dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35975 } } }
I20260928 05:17:59.911278 8524 tablet_replica.cc:442] T ea5bb4026f664d0fab67f0e7e7c70f5e P c35d4d4f8f3a49a09bfaec1f37147dd2: This TxnStatusTablet replica's current role is: LEADER
I20260928 05:17:59.911298 8483 tablet_replica.cc:439] T ea5bb4026f664d0fab67f0e7e7c70f5e P eeb06b1625b4449da5c75ca573e0f3e6: TxnStatusTablet state changed. Reason: New leader c35d4d4f8f3a49a09bfaec1f37147dd2. Latest consensus state: current_term: 1 leader_uuid: "c35d4d4f8f3a49a09bfaec1f37147dd2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f89f414d76944cfb5a9505edaa66d35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42423 } } peers { permanent_uuid: "eeb06b1625b4449da5c75ca573e0f3e6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44359 } } peers { permanent_uuid: "c35d4d4f8f3a49a09bfaec1f37147dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35975 } } }
I20260928 05:17:59.911573 8483 tablet_replica.cc:442] T ea5bb4026f664d0fab67f0e7e7c70f5e P eeb06b1625b4449da5c75ca573e0f3e6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 05:17:59.913437 8473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:59.906218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43430 (local address 127.0.97.3:35975)
0928 05:17:59.906459 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:59.906464 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:59.906480 (+ 16us) server_negotiation.cc:407] Connection header received
0928 05:17:59.906553 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:59.906557 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:59.906642 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:59.906718 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:17:59.907181 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:59.908148 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:59.909943 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:59.910562 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:59.910680 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:59.912473 (+ 1793us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:17:59.912590 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:17:59.912663 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:59.913093 (+ 430us) server_negotiation.cc:299] Negotiation successful
0928 05:17:59.913169 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144}
I20260928 05:17:59.915052 8490 mvcc.cc:205] Tried to move back new op lower bound from 7334185696824958976 to 7334185696732229632. Current Snapshot: MvccSnapshot[applied={T|T < 7334185696824958976}]
I20260928 05:17:59.915721 8483 tablet_replica.cc:439] T ea5bb4026f664d0fab67f0e7e7c70f5e P eeb06b1625b4449da5c75ca573e0f3e6: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c35d4d4f8f3a49a09bfaec1f37147dd2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f89f414d76944cfb5a9505edaa66d35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42423 } } peers { permanent_uuid: "eeb06b1625b4449da5c75ca573e0f3e6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44359 } } peers { permanent_uuid: "c35d4d4f8f3a49a09bfaec1f37147dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35975 } } }
I20260928 05:17:59.915932 8483 tablet_replica.cc:442] T ea5bb4026f664d0fab67f0e7e7c70f5e P eeb06b1625b4449da5c75ca573e0f3e6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 05:17:59.907272 8527 tablet_replica.cc:439] T ea5bb4026f664d0fab67f0e7e7c70f5e P 0f89f414d76944cfb5a9505edaa66d35: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c35d4d4f8f3a49a09bfaec1f37147dd2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f89f414d76944cfb5a9505edaa66d35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42423 } } peers { permanent_uuid: "eeb06b1625b4449da5c75ca573e0f3e6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44359 } } peers { permanent_uuid: "c35d4d4f8f3a49a09bfaec1f37147dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35975 } } }
I20260928 05:17:59.919447 8527 tablet_replica.cc:442] T ea5bb4026f664d0fab67f0e7e7c70f5e P 0f89f414d76944cfb5a9505edaa66d35: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 05:17:59.930963 8473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:59.925721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46457 (local address 127.0.97.3:35975)
0928 05:17:59.925890 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 05:17:59.925896 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:17:59.926502 (+ 606us) server_negotiation.cc:407] Connection header received
0928 05:17:59.926673 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:17:59.926678 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:17:59.926733 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:17:59.926798 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:17:59.927264 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:59.927973 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:59.929359 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:17:59.929788 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:17:59.929889 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:59.930253 (+ 364us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:17:59.930372 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:17:59.930438 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 05:17:59.930571 (+ 133us) server_negotiation.cc:299] Negotiation successful
0928 05:17:59.930635 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 05:17:59.931047 8530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:17:59.925735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35975 (local address 127.0.97.3:46457)
0928 05:17:59.926416 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0928 05:17:59.926430 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 05:17:59.926562 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:17:59.926839 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:17:59.926844 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:17:59.926853 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 05:17:59.927131 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:59.927139 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:59.928088 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:59.928097 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:59.929243 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:17:59.929254 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:17:59.929949 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:17:59.929955 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:17:59.930066 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:17:59.930111 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 05:17:59.930456 (+ 345us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 05:17:59.930478 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 05:17:59.930550 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 05:17:59.930668 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":391,"threads_started":1}
I20260928 05:17:59.940879 8268 raft_consensus.cc:1261] T f07bad35e54046afb27621bb02449472 P eeb06b1625b4449da5c75ca573e0f3e6 [term 1 FOLLOWER]: Refusing update from remote peer c35d4d4f8f3a49a09bfaec1f37147dd2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:17:59.941453 8484 consensus_queue.cc:1035] T f07bad35e54046afb27621bb02449472 P c35d4d4f8f3a49a09bfaec1f37147dd2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeb06b1625b4449da5c75ca573e0f3e6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:17:59.949096 8124 raft_consensus.cc:1261] T f07bad35e54046afb27621bb02449472 P 0f89f414d76944cfb5a9505edaa66d35 [term 1 FOLLOWER]: Refusing update from remote peer c35d4d4f8f3a49a09bfaec1f37147dd2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:17:59.949611 8484 consensus_queue.cc:1035] T f07bad35e54046afb27621bb02449472 P c35d4d4f8f3a49a09bfaec1f37147dd2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f89f414d76944cfb5a9505edaa66d35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:18:00.020653 7966 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53242:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:18:00.020931 7966 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53242:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:18:00.023950 7966 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ca2877e519c94fd181176dfe63838bb3: Sending DeleteTablet for 3 replicas of tablet f07bad35e54046afb27621bb02449472
I20260928 05:18:00.024878 8242 tablet_service.cc:1514] Processing DeleteTablet for tablet f07bad35e54046afb27621bb02449472 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:18:00 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47436
I20260928 05:18:00.025128 8501 tablet_replica.cc:330] T f07bad35e54046afb27621bb02449472 P eeb06b1625b4449da5c75ca573e0f3e6: stopping tablet replica
I20260928 05:18:00.025341 8501 raft_consensus.cc:2229] T f07bad35e54046afb27621bb02449472 P eeb06b1625b4449da5c75ca573e0f3e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:18:00.025573 8501 raft_consensus.cc:2258] T f07bad35e54046afb27621bb02449472 P eeb06b1625b4449da5c75ca573e0f3e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:18:00.025555 8104 tablet_service.cc:1514] Processing DeleteTablet for tablet f07bad35e54046afb27621bb02449472 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:18:00 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46714
I20260928 05:18:00.025815 8502 tablet_replica.cc:330] T f07bad35e54046afb27621bb02449472 P 0f89f414d76944cfb5a9505edaa66d35: stopping tablet replica
I20260928 05:18:00.026010 8502 raft_consensus.cc:2229] T f07bad35e54046afb27621bb02449472 P 0f89f414d76944cfb5a9505edaa66d35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:18:00.026180 8502 raft_consensus.cc:2258] T f07bad35e54046afb27621bb02449472 P 0f89f414d76944cfb5a9505edaa66d35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:18:00.026671 8501 ts_tablet_manager.cc:1905] T f07bad35e54046afb27621bb02449472 P eeb06b1625b4449da5c75ca573e0f3e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:18:00.027254 8502 ts_tablet_manager.cc:1905] T f07bad35e54046afb27621bb02449472 P 0f89f414d76944cfb5a9505edaa66d35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:18:00.029075 8382 tablet_service.cc:1514] Processing DeleteTablet for tablet f07bad35e54046afb27621bb02449472 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:18:00 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43410
I20260928 05:18:00.029376 8500 tablet_replica.cc:330] T f07bad35e54046afb27621bb02449472 P c35d4d4f8f3a49a09bfaec1f37147dd2: stopping tablet replica
I20260928 05:18:00.029557 8500 raft_consensus.cc:2229] T f07bad35e54046afb27621bb02449472 P c35d4d4f8f3a49a09bfaec1f37147dd2 [term 1 LEADER]: Raft consensus shutting down.
I20260928 05:18:00.029997 8500 raft_consensus.cc:2258] T f07bad35e54046afb27621bb02449472 P c35d4d4f8f3a49a09bfaec1f37147dd2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:18:00.030978 8500 ts_tablet_manager.cc:1905] T f07bad35e54046afb27621bb02449472 P c35d4d4f8f3a49a09bfaec1f37147dd2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:18:00.033210 8501 ts_tablet_manager.cc:1918] T f07bad35e54046afb27621bb02449472 P eeb06b1625b4449da5c75ca573e0f3e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 05:18:00.033352 8501 log.cc:1196] T f07bad35e54046afb27621bb02449472 P eeb06b1625b4449da5c75ca573e0f3e6: Deleting WAL directory at /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/minicluster-data/ts-1/wal/wals/f07bad35e54046afb27621bb02449472
I20260928 05:18:00.033464 8502 ts_tablet_manager.cc:1918] T f07bad35e54046afb27621bb02449472 P 0f89f414d76944cfb5a9505edaa66d35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 05:18:00.033603 8502 log.cc:1196] T f07bad35e54046afb27621bb02449472 P 0f89f414d76944cfb5a9505edaa66d35: Deleting WAL directory at /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/minicluster-data/ts-0/wal/wals/f07bad35e54046afb27621bb02449472
I20260928 05:18:00.033710 8501 ts_tablet_manager.cc:1939] T f07bad35e54046afb27621bb02449472 P eeb06b1625b4449da5c75ca573e0f3e6: Deleting consensus metadata
I20260928 05:18:00.033968 8502 ts_tablet_manager.cc:1939] T f07bad35e54046afb27621bb02449472 P 0f89f414d76944cfb5a9505edaa66d35: Deleting consensus metadata
I20260928 05:18:00.034729 7951 catalog_manager.cc:4841] TS eeb06b1625b4449da5c75ca573e0f3e6 (127.0.97.2:44359): tablet f07bad35e54046afb27621bb02449472 (table test-table [id=b5792c948c2745968bc0ef8e930b78ed]) successfully deleted
I20260928 05:18:00.034838 7954 catalog_manager.cc:4841] TS 0f89f414d76944cfb5a9505edaa66d35 (127.0.97.1:42423): tablet f07bad35e54046afb27621bb02449472 (table test-table [id=b5792c948c2745968bc0ef8e930b78ed]) successfully deleted
I20260928 05:18:00.037464 8500 ts_tablet_manager.cc:1918] T f07bad35e54046afb27621bb02449472 P c35d4d4f8f3a49a09bfaec1f37147dd2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 05:18:00.037600 8500 log.cc:1196] T f07bad35e54046afb27621bb02449472 P c35d4d4f8f3a49a09bfaec1f37147dd2: Deleting WAL directory at /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/minicluster-data/ts-2/wal/wals/f07bad35e54046afb27621bb02449472
I20260928 05:18:00.038025 8500 ts_tablet_manager.cc:1939] T f07bad35e54046afb27621bb02449472 P c35d4d4f8f3a49a09bfaec1f37147dd2: Deleting consensus metadata
I20260928 05:18:00.038913 7953 catalog_manager.cc:4841] TS c35d4d4f8f3a49a09bfaec1f37147dd2 (127.0.97.3:35975): tablet f07bad35e54046afb27621bb02449472 (table test-table [id=b5792c948c2745968bc0ef8e930b78ed]) successfully deleted
Sep 28 05:18:00 dist-test-slave-whsn krb5kdc[7922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 05:18:00 dist-test-slave-whsn krb5kdc[7922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:18:00 dist-test-slave-whsn krb5kdc[7922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 05:18:00.089833 8516 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:18:00.077863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53260 (local address 127.0.97.62:39261)
0928 05:18:00.078088 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:00.078096 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:00.081324 (+ 3228us) server_negotiation.cc:407] Connection header received
0928 05:18:00.081508 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:00.081513 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:00.081575 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:00.081655 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:18:00.082845 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:00.083552 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:00.084772 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:00.085242 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:00.085349 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:00.089664 (+ 4315us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53260: BlockingRecv error: recv got EOF from 127.0.0.1:53260 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 28 05:18:00 dist-test-slave-whsn krb5kdc[7922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260928 05:18:00.105669 8516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:00.093179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53274 (local address 127.0.97.62:39261)
0928 05:18:00.093321 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:00.093327 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:00.093828 (+ 501us) server_negotiation.cc:407] Connection header received
0928 05:18:00.093970 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:00.093975 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:00.094038 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:00.094115 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:18:00.095175 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:00.095876 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:00.097130 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:00.097534 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:00.097650 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:00.100961 (+ 3311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:18:00.100965 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:18:00.100968 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:18:00.100998 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:18:00.103298 (+ 2300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:18:00.103859 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:18:00.103865 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:18:00.103867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:18:00.103989 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:18:00.104320 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:18:00.104325 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:18:00.104328 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:18:00.104567 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:18:00.104804 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0928 05:18:00.105140 (+ 336us) server_negotiation.cc:299] Negotiation successful
0928 05:18:00.105340 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 05:18:00.108688 7966 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53274:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 05:18:00.109249 7966 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 05:18:00.115442 8104 tablet_service.cc:1467] Processing CreateTablet for tablet 3b7f9bbd946d4fcab6b9796a9ff0fdfc (DEFAULT_TABLE table=test-table [id=64edd173c7804868b45d00a2f27065e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:18:00.115546 8382 tablet_service.cc:1467] Processing CreateTablet for tablet 3b7f9bbd946d4fcab6b9796a9ff0fdfc (DEFAULT_TABLE table=test-table [id=64edd173c7804868b45d00a2f27065e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:18:00.115880 8104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b7f9bbd946d4fcab6b9796a9ff0fdfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:18:00.115962 8382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b7f9bbd946d4fcab6b9796a9ff0fdfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:18:00.117208 8242 tablet_service.cc:1467] Processing CreateTablet for tablet 3b7f9bbd946d4fcab6b9796a9ff0fdfc (DEFAULT_TABLE table=test-table [id=64edd173c7804868b45d00a2f27065e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:18:00.117619 8242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b7f9bbd946d4fcab6b9796a9ff0fdfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:18:00.121673 8477 tablet_bootstrap.cc:492] T 3b7f9bbd946d4fcab6b9796a9ff0fdfc P 0f89f414d76944cfb5a9505edaa66d35: Bootstrap starting.
I20260928 05:18:00.122804 8478 tablet_bootstrap.cc:492] T 3b7f9bbd946d4fcab6b9796a9ff0fdfc P c35d4d4f8f3a49a09bfaec1f37147dd2: Bootstrap starting.
I20260928 05:18:00.123229 8477 tablet_bootstrap.cc:654] T 3b7f9bbd946d4fcab6b9796a9ff0fdfc P 0f89f414d76944cfb5a9505edaa66d35: Neither blocks nor log segments found. Creating new log.
I20260928 05:18:00.124360 8478 tablet_bootstrap.cc:654] T 3b7f9bbd946d4fcab6b9796a9ff0fdfc P c35d4d4f8f3a49a09bfaec1f37147dd2: Neither blocks nor log segments found. Creating new log.
I20260928 05:18:00.126276 8477 tablet_bootstrap.cc:492] T 3b7f9bbd946d4fcab6b9796a9ff0fdfc P 0f89f414d76944cfb5a9505edaa66d35: No bootstrap required, opened a new log
I20260928 05:18:00.126371 8476 tablet_bootstrap.cc:492] T 3b7f9bbd946d4fcab6b9796a9ff0fdfc P eeb06b1625b4449da5c75ca573e0f3e6: Bootstrap starting.
I20260928 05:18:00.126410 8477 ts_tablet_manager.cc:1397] T 3b7f9bbd946d4fcab6b9796a9ff0fdfc P 0f89f414d76944cfb5a9505edaa66d35: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260928 05:18:00.127116 8477 raft_consensus.cc:348] T 3b7f9bbd946d4fcab6b9796a9ff0fdfc P 0f89f414d76944cfb5a9505edaa66d35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f89f414d76944cfb5a9505edaa66d35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42423 } } peers { permanent_uuid: "eeb06b1625b4449da5c75ca573e0f3e6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44359 } } peers { permanent_uuid: "c35d4d4f8f3a49a09bfaec1f37147dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35975 } }
I20260928 05:18:00.127341 8477 raft_consensus.cc:374] T 3b7f9bbd946d4fcab6b9796a9ff0fdfc P 0f89f414d76944cfb5a9505edaa66d35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:18:00.127417 8477 raft_consensus.cc:729] T 3b7f9bbd946d4fcab6b9796a9ff0fdfc P 0f89f414d76944cfb5a9505edaa66d35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f89f414d76944cfb5a9505edaa66d35, State: Initialized, Role: FOLLOWER
I20260928 05:18:00.127653 8476 tablet_bootstrap.cc:654] T 3b7f9bbd946d4fcab6b9796a9ff0fdfc P eeb06b1625b4449da5c75ca573e0f3e6: Neither blocks nor log segments found. Creating new log.
I20260928 05:18:00.127621 8477 consensus_queue.cc:260] T 3b7f9bbd946d4fcab6b9796a9ff0fdfc P 0f89f414d76944cfb5a9505edaa66d35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f89f414d76944cfb5a9505edaa66d35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42423 } } peers { permanent_uuid: "eeb06b1625b4449da5c75ca573e0f3e6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44359 } } peers { permanent_uuid: "c35d4d4f8f3a49a09bfaec1f37147dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35975 } }
I20260928 05:18:00.128139 8477 ts_tablet_manager.cc:1428] T 3b7f9bbd946d4fcab6b9796a9ff0fdfc P 0f89f414d76944cfb5a9505edaa66d35: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260928 05:18:00.130004 8476 tablet_bootstrap.cc:492] T 3b7f9bbd946d4fcab6b9796a9ff0fdfc P eeb06b1625b4449da5c75ca573e0f3e6: No bootstrap required, opened a new log
I20260928 05:18:00.130142 8476 ts_tablet_manager.cc:1397] T 3b7f9bbd946d4fcab6b9796a9ff0fdfc P eeb06b1625b4449da5c75ca573e0f3e6: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260928 05:18:00.130846 8476 raft_consensus.cc:348] T 3b7f9bbd946d4fcab6b9796a9ff0fdfc P eeb06b1625b4449da5c75ca573e0f3e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f89f414d76944cfb5a9505edaa66d35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42423 } } peers { permanent_uuid: "eeb06b1625b4449da5c75ca573e0f3e6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44359 } } peers { permanent_uuid: "c35d4d4f8f3a49a09bfaec1f37147dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35975 } }
I20260928 05:18:00.131095 8476 raft_consensus.cc:374] T 3b7f9bbd946d4fcab6b9796a9ff0fdfc P eeb06b1625b4449da5c75ca573e0f3e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:18:00.131196 8476 raft_consensus.cc:729] T 3b7f9bbd946d4fcab6b9796a9ff0fdfc P eeb06b1625b4449da5c75ca573e0f3e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeb06b1625b4449da5c75ca573e0f3e6, State: Initialized, Role: FOLLOWER
I20260928 05:18:00.131402 8476 consensus_queue.cc:260] T 3b7f9bbd946d4fcab6b9796a9ff0fdfc P eeb06b1625b4449da5c75ca573e0f3e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f89f414d76944cfb5a9505edaa66d35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42423 } } peers { permanent_uuid: "eeb06b1625b4449da5c75ca573e0f3e6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44359 } } peers { permanent_uuid: "c35d4d4f8f3a49a09bfaec1f37147dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35975 } }
I20260928 05:18:00.131907 8476 ts_tablet_manager.cc:1428] T 3b7f9bbd946d4fcab6b9796a9ff0fdfc P eeb06b1625b4449da5c75ca573e0f3e6: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260928 05:18:00.133824 8478 tablet_bootstrap.cc:492] T 3b7f9bbd946d4fcab6b9796a9ff0fdfc P c35d4d4f8f3a49a09bfaec1f37147dd2: No bootstrap required, opened a new log
I20260928 05:18:00.133934 8478 ts_tablet_manager.cc:1397] T 3b7f9bbd946d4fcab6b9796a9ff0fdfc P c35d4d4f8f3a49a09bfaec1f37147dd2: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.003s
I20260928 05:18:00.134569 8478 raft_consensus.cc:348] T 3b7f9bbd946d4fcab6b9796a9ff0fdfc P c35d4d4f8f3a49a09bfaec1f37147dd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f89f414d76944cfb5a9505edaa66d35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42423 } } peers { permanent_uuid: "eeb06b1625b4449da5c75ca573e0f3e6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44359 } } peers { permanent_uuid: "c35d4d4f8f3a49a09bfaec1f37147dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35975 } }
I20260928 05:18:00.134747 8478 raft_consensus.cc:374] T 3b7f9bbd946d4fcab6b9796a9ff0fdfc P c35d4d4f8f3a49a09bfaec1f37147dd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:18:00.134807 8478 raft_consensus.cc:729] T 3b7f9bbd946d4fcab6b9796a9ff0fdfc P c35d4d4f8f3a49a09bfaec1f37147dd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c35d4d4f8f3a49a09bfaec1f37147dd2, State: Initialized, Role: FOLLOWER
I20260928 05:18:00.134982 8478 consensus_queue.cc:260] T 3b7f9bbd946d4fcab6b9796a9ff0fdfc P c35d4d4f8f3a49a09bfaec1f37147dd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f89f414d76944cfb5a9505edaa66d35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42423 } } peers { permanent_uuid: "eeb06b1625b4449da5c75ca573e0f3e6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44359 } } peers { permanent_uuid: "c35d4d4f8f3a49a09bfaec1f37147dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35975 } }
I20260928 05:18:00.136006 8478 ts_tablet_manager.cc:1428] T 3b7f9bbd946d4fcab6b9796a9ff0fdfc P c35d4d4f8f3a49a09bfaec1f37147dd2: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260928 05:18:00.232820 8527 raft_consensus.cc:482] T 3b7f9bbd946d4fcab6b9796a9ff0fdfc P 0f89f414d76944cfb5a9505edaa66d35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:18:00.233029 8527 raft_consensus.cc:504] T 3b7f9bbd946d4fcab6b9796a9ff0fdfc P 0f89f414d76944cfb5a9505edaa66d35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f89f414d76944cfb5a9505edaa66d35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42423 } } peers { permanent_uuid: "eeb06b1625b4449da5c75ca573e0f3e6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44359 } } peers { permanent_uuid: "c35d4d4f8f3a49a09bfaec1f37147dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35975 } }
I20260928 05:18:00.234448 8527 leader_election.cc:288] T 3b7f9bbd946d4fcab6b9796a9ff0fdfc P 0f89f414d76944cfb5a9505edaa66d35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eeb06b1625b4449da5c75ca573e0f3e6 (127.0.97.2:44359), c35d4d4f8f3a49a09bfaec1f37147dd2 (127.0.97.3:35975)
I20260928 05:18:00.251744 8552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:00.234792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35357 (local address 127.0.97.2:44359)
0928 05:18:00.246010 (+ 11218us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:00.246016 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:00.246038 (+ 22us) server_negotiation.cc:407] Connection header received
0928 05:18:00.246109 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:00.246114 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:00.246187 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:00.246933 (+ 746us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:18:00.247116 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:00.248220 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:00.250068 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:00.250777 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:00.250885 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:00.250978 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 05:18:00.251326 (+ 348us) server_negotiation.cc:299] Negotiation successful
0928 05:18:00.251453 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11082,"thread_start_us":11228,"threads_started":1}
I20260928 05:18:00.251791 8553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:00.234706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44359 (local address 127.0.97.1:35357)
0928 05:18:00.245744 (+ 11038us) negotiation.cc:107] Waiting for socket to connect
0928 05:18:00.245767 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 05:18:00.245919 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:18:00.246423 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:18:00.246429 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:18:00.246458 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:18:00.246773 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:00.246786 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:00.248364 (+ 1578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:00.248369 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:00.249918 (+ 1549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:00.249928 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:00.250919 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:00.250925 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:00.251162 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:00.251209 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 05:18:00.251286 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 05:18:00.251404 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10906,"mutex_wait_us":79,"thread_start_us":11561,"threads_started":1}
I20260928 05:18:00.252470 8267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b7f9bbd946d4fcab6b9796a9ff0fdfc" candidate_uuid: "0f89f414d76944cfb5a9505edaa66d35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeb06b1625b4449da5c75ca573e0f3e6" is_pre_election: true
I20260928 05:18:00.252712 8267 raft_consensus.cc:2454] T 3b7f9bbd946d4fcab6b9796a9ff0fdfc P eeb06b1625b4449da5c75ca573e0f3e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f89f414d76944cfb5a9505edaa66d35 in term 0.
I20260928 05:18:00.253162 8054 leader_election.cc:302] T 3b7f9bbd946d4fcab6b9796a9ff0fdfc P 0f89f414d76944cfb5a9505edaa66d35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f89f414d76944cfb5a9505edaa66d35, eeb06b1625b4449da5c75ca573e0f3e6; no voters:
I20260928 05:18:00.253594 8527 raft_consensus.cc:2781] T 3b7f9bbd946d4fcab6b9796a9ff0fdfc P 0f89f414d76944cfb5a9505edaa66d35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 05:18:00.253714 8527 raft_consensus.cc:482] T 3b7f9bbd946d4fcab6b9796a9ff0fdfc P 0f89f414d76944cfb5a9505edaa66d35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 05:18:00.253788 8527 raft_consensus.cc:3037] T 3b7f9bbd946d4fcab6b9796a9ff0fdfc P 0f89f414d76944cfb5a9505edaa66d35 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:18:00.253829 8473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:00.235263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49475 (local address 127.0.97.3:35975)
0928 05:18:00.235461 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:00.235470 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:00.246829 (+ 11359us) server_negotiation.cc:407] Connection header received
0928 05:18:00.247091 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:00.247096 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:00.247170 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:00.247231 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:18:00.247923 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:00.248876 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:00.251368 (+ 2492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:00.252618 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:00.252747 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:00.252881 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 05:18:00.253425 (+ 544us) server_negotiation.cc:299] Negotiation successful
0928 05:18:00.253557 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 05:18:00.254562 8551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:00.234805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35975 (local address 127.0.97.1:49475)
0928 05:18:00.246739 (+ 11934us) negotiation.cc:107] Waiting for socket to connect
0928 05:18:00.246758 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 05:18:00.246915 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:18:00.247452 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:18:00.247456 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:18:00.247479 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:18:00.247776 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:00.247784 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:00.249025 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:00.249031 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:00.251203 (+ 2172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:00.251218 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:00.253042 (+ 1824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:00.253049 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:00.253235 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:00.253271 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 05:18:00.254026 (+ 755us) client_negotiation.cc:240] Negotiation successful
0928 05:18:00.254154 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11818,"mutex_wait_us":60,"thread_start_us":10687,"threads_started":1}
I20260928 05:18:00.255183 8407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b7f9bbd946d4fcab6b9796a9ff0fdfc" candidate_uuid: "0f89f414d76944cfb5a9505edaa66d35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c35d4d4f8f3a49a09bfaec1f37147dd2" is_pre_election: true
I20260928 05:18:00.255420 8407 raft_consensus.cc:2454] T 3b7f9bbd946d4fcab6b9796a9ff0fdfc P c35d4d4f8f3a49a09bfaec1f37147dd2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f89f414d76944cfb5a9505edaa66d35 in term 0.
I20260928 05:18:00.257052 8527 raft_consensus.cc:504] T 3b7f9bbd946d4fcab6b9796a9ff0fdfc P 0f89f414d76944cfb5a9505edaa66d35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f89f414d76944cfb5a9505edaa66d35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42423 } } peers { permanent_uuid: "eeb06b1625b4449da5c75ca573e0f3e6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44359 } } peers { permanent_uuid: "c35d4d4f8f3a49a09bfaec1f37147dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35975 } }
I20260928 05:18:00.257779 8527 leader_election.cc:288] T 3b7f9bbd946d4fcab6b9796a9ff0fdfc P 0f89f414d76944cfb5a9505edaa66d35 [CANDIDATE]: Term 1 election: Requested vote from peers eeb06b1625b4449da5c75ca573e0f3e6 (127.0.97.2:44359), c35d4d4f8f3a49a09bfaec1f37147dd2 (127.0.97.3:35975)
I20260928 05:18:00.258141 8267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b7f9bbd946d4fcab6b9796a9ff0fdfc" candidate_uuid: "0f89f414d76944cfb5a9505edaa66d35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeb06b1625b4449da5c75ca573e0f3e6"
I20260928 05:18:00.258319 8267 raft_consensus.cc:3037] T 3b7f9bbd946d4fcab6b9796a9ff0fdfc P eeb06b1625b4449da5c75ca573e0f3e6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:18:00.259143 8407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b7f9bbd946d4fcab6b9796a9ff0fdfc" candidate_uuid: "0f89f414d76944cfb5a9505edaa66d35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c35d4d4f8f3a49a09bfaec1f37147dd2"
I20260928 05:18:00.259356 8407 raft_consensus.cc:3037] T 3b7f9bbd946d4fcab6b9796a9ff0fdfc P c35d4d4f8f3a49a09bfaec1f37147dd2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:18:00.260536 8267 raft_consensus.cc:2454] T 3b7f9bbd946d4fcab6b9796a9ff0fdfc P eeb06b1625b4449da5c75ca573e0f3e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f89f414d76944cfb5a9505edaa66d35 in term 1.
I20260928 05:18:00.260914 8054 leader_election.cc:302] T 3b7f9bbd946d4fcab6b9796a9ff0fdfc P 0f89f414d76944cfb5a9505edaa66d35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f89f414d76944cfb5a9505edaa66d35, eeb06b1625b4449da5c75ca573e0f3e6; no voters:
I20260928 05:18:00.261164 8527 raft_consensus.cc:2781] T 3b7f9bbd946d4fcab6b9796a9ff0fdfc P 0f89f414d76944cfb5a9505edaa66d35 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:18:00.261377 8527 raft_consensus.cc:686] T 3b7f9bbd946d4fcab6b9796a9ff0fdfc P 0f89f414d76944cfb5a9505edaa66d35 [term 1 LEADER]: Becoming Leader. State: Replica: 0f89f414d76944cfb5a9505edaa66d35, State: Running, Role: LEADER
I20260928 05:18:00.261719 8527 consensus_queue.cc:237] T 3b7f9bbd946d4fcab6b9796a9ff0fdfc P 0f89f414d76944cfb5a9505edaa66d35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f89f414d76944cfb5a9505edaa66d35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42423 } } peers { permanent_uuid: "eeb06b1625b4449da5c75ca573e0f3e6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44359 } } peers { permanent_uuid: "c35d4d4f8f3a49a09bfaec1f37147dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35975 } }
I20260928 05:18:00.262508 8407 raft_consensus.cc:2454] T 3b7f9bbd946d4fcab6b9796a9ff0fdfc P c35d4d4f8f3a49a09bfaec1f37147dd2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f89f414d76944cfb5a9505edaa66d35 in term 1.
I20260928 05:18:00.265178 7967 catalog_manager.cc:5495] T 3b7f9bbd946d4fcab6b9796a9ff0fdfc P 0f89f414d76944cfb5a9505edaa66d35 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0f89f414d76944cfb5a9505edaa66d35 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "0f89f414d76944cfb5a9505edaa66d35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f89f414d76944cfb5a9505edaa66d35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42423 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eeb06b1625b4449da5c75ca573e0f3e6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c35d4d4f8f3a49a09bfaec1f37147dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35975 } health_report { overall_health: UNKNOWN } } }
I20260928 05:18:00.287685 8556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:00.281447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46718 (local address 127.0.97.1:42423)
0928 05:18:00.282004 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:00.282010 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:00.282034 (+ 24us) server_negotiation.cc:407] Connection header received
0928 05:18:00.282127 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:00.282132 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:00.282226 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:00.282404 (+ 178us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:18:00.282667 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:00.283824 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:00.285103 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:00.285719 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:00.285849 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:00.286188 (+ 339us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:18:00.286996 (+ 808us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:18:00.287169 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0928 05:18:00.287224 (+ 55us) server_negotiation.cc:299] Negotiation successful
0928 05:18:00.287299 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":296,"threads_started":1}
I20260928 05:18:00.291144 8267 raft_consensus.cc:1261] T 3b7f9bbd946d4fcab6b9796a9ff0fdfc P eeb06b1625b4449da5c75ca573e0f3e6 [term 1 FOLLOWER]: Refusing update from remote peer 0f89f414d76944cfb5a9505edaa66d35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:18:00.291167 8407 raft_consensus.cc:1261] T 3b7f9bbd946d4fcab6b9796a9ff0fdfc P c35d4d4f8f3a49a09bfaec1f37147dd2 [term 1 FOLLOWER]: Refusing update from remote peer 0f89f414d76944cfb5a9505edaa66d35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:18:00.291823 8482 consensus_queue.cc:1035] T 3b7f9bbd946d4fcab6b9796a9ff0fdfc P 0f89f414d76944cfb5a9505edaa66d35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeb06b1625b4449da5c75ca573e0f3e6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:18:00.292099 8527 consensus_queue.cc:1035] T 3b7f9bbd946d4fcab6b9796a9ff0fdfc P 0f89f414d76944cfb5a9505edaa66d35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c35d4d4f8f3a49a09bfaec1f37147dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:18:00.315196 7966 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53274:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:18:00.315433 7966 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53274:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:18:00.318534 7966 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ca2877e519c94fd181176dfe63838bb3: Sending DeleteTablet for 3 replicas of tablet 3b7f9bbd946d4fcab6b9796a9ff0fdfc
I20260928 05:18:00.319207 8104 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b7f9bbd946d4fcab6b9796a9ff0fdfc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:18:00 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46714
I20260928 05:18:00.319401 8242 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b7f9bbd946d4fcab6b9796a9ff0fdfc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:18:00 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47436
I20260928 05:18:00.319478 8502 tablet_replica.cc:330] T 3b7f9bbd946d4fcab6b9796a9ff0fdfc P 0f89f414d76944cfb5a9505edaa66d35: stopping tablet replica
I20260928 05:18:00.319670 8501 tablet_replica.cc:330] T 3b7f9bbd946d4fcab6b9796a9ff0fdfc P eeb06b1625b4449da5c75ca573e0f3e6: stopping tablet replica
I20260928 05:18:00.319654 8502 raft_consensus.cc:2229] T 3b7f9bbd946d4fcab6b9796a9ff0fdfc P 0f89f414d76944cfb5a9505edaa66d35 [term 1 LEADER]: Raft consensus shutting down.
I20260928 05:18:00.319818 8501 raft_consensus.cc:2229] T 3b7f9bbd946d4fcab6b9796a9ff0fdfc P eeb06b1625b4449da5c75ca573e0f3e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:18:00.320016 8501 raft_consensus.cc:2258] T 3b7f9bbd946d4fcab6b9796a9ff0fdfc P eeb06b1625b4449da5c75ca573e0f3e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:18:00.320106 8502 raft_consensus.cc:2258] T 3b7f9bbd946d4fcab6b9796a9ff0fdfc P 0f89f414d76944cfb5a9505edaa66d35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:18:00.320931 8382 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b7f9bbd946d4fcab6b9796a9ff0fdfc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:18:00 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43410
I20260928 05:18:00.321189 8500 tablet_replica.cc:330] T 3b7f9bbd946d4fcab6b9796a9ff0fdfc P c35d4d4f8f3a49a09bfaec1f37147dd2: stopping tablet replica
I20260928 05:18:00.321357 8500 raft_consensus.cc:2229] T 3b7f9bbd946d4fcab6b9796a9ff0fdfc P c35d4d4f8f3a49a09bfaec1f37147dd2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:18:00.321593 8500 raft_consensus.cc:2258] T 3b7f9bbd946d4fcab6b9796a9ff0fdfc P c35d4d4f8f3a49a09bfaec1f37147dd2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:18:00.322712 7966 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53274:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 05:18:00.322945 8501 ts_tablet_manager.cc:1905] T 3b7f9bbd946d4fcab6b9796a9ff0fdfc P eeb06b1625b4449da5c75ca573e0f3e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:18:00.323210 8500 ts_tablet_manager.cc:1905] T 3b7f9bbd946d4fcab6b9796a9ff0fdfc P c35d4d4f8f3a49a09bfaec1f37147dd2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 05:18:00.323308 7966 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 05:18:00.326460 8502 ts_tablet_manager.cc:1905] T 3b7f9bbd946d4fcab6b9796a9ff0fdfc P 0f89f414d76944cfb5a9505edaa66d35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:18:00.328339 8500 ts_tablet_manager.cc:1918] T 3b7f9bbd946d4fcab6b9796a9ff0fdfc P c35d4d4f8f3a49a09bfaec1f37147dd2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:18:00.328449 8500 log.cc:1196] T 3b7f9bbd946d4fcab6b9796a9ff0fdfc P c35d4d4f8f3a49a09bfaec1f37147dd2: Deleting WAL directory at /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/minicluster-data/ts-2/wal/wals/3b7f9bbd946d4fcab6b9796a9ff0fdfc
I20260928 05:18:00.328775 8500 ts_tablet_manager.cc:1939] T 3b7f9bbd946d4fcab6b9796a9ff0fdfc P c35d4d4f8f3a49a09bfaec1f37147dd2: Deleting consensus metadata
I20260928 05:18:00.329898 7953 catalog_manager.cc:4841] TS c35d4d4f8f3a49a09bfaec1f37147dd2 (127.0.97.3:35975): tablet 3b7f9bbd946d4fcab6b9796a9ff0fdfc (table test-table [id=64edd173c7804868b45d00a2f27065e6]) successfully deleted
I20260928 05:18:00.332984 8502 ts_tablet_manager.cc:1918] T 3b7f9bbd946d4fcab6b9796a9ff0fdfc P 0f89f414d76944cfb5a9505edaa66d35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:18:00.333103 8502 log.cc:1196] T 3b7f9bbd946d4fcab6b9796a9ff0fdfc P 0f89f414d76944cfb5a9505edaa66d35: Deleting WAL directory at /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/minicluster-data/ts-0/wal/wals/3b7f9bbd946d4fcab6b9796a9ff0fdfc
I20260928 05:18:00.333509 8502 ts_tablet_manager.cc:1939] T 3b7f9bbd946d4fcab6b9796a9ff0fdfc P 0f89f414d76944cfb5a9505edaa66d35: Deleting consensus metadata
I20260928 05:18:00.334518 7954 catalog_manager.cc:4841] TS 0f89f414d76944cfb5a9505edaa66d35 (127.0.97.1:42423): tablet 3b7f9bbd946d4fcab6b9796a9ff0fdfc (table test-table [id=64edd173c7804868b45d00a2f27065e6]) successfully deleted
I20260928 05:18:00.338223 8242 tablet_service.cc:1467] Processing CreateTablet for tablet c1297cecd1ef43e386aeec2eb0fd3945 (DEFAULT_TABLE table=test-table [id=f60060ca1ab844e5b57742bf01617411]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:18:00.338625 8242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1297cecd1ef43e386aeec2eb0fd3945. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:18:00.339219 8501 ts_tablet_manager.cc:1918] T 3b7f9bbd946d4fcab6b9796a9ff0fdfc P eeb06b1625b4449da5c75ca573e0f3e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:18:00.339413 8501 log.cc:1196] T 3b7f9bbd946d4fcab6b9796a9ff0fdfc P eeb06b1625b4449da5c75ca573e0f3e6: Deleting WAL directory at /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/minicluster-data/ts-1/wal/wals/3b7f9bbd946d4fcab6b9796a9ff0fdfc
I20260928 05:18:00.339917 8501 ts_tablet_manager.cc:1939] T 3b7f9bbd946d4fcab6b9796a9ff0fdfc P eeb06b1625b4449da5c75ca573e0f3e6: Deleting consensus metadata
I20260928 05:18:00.341109 7951 catalog_manager.cc:4841] TS eeb06b1625b4449da5c75ca573e0f3e6 (127.0.97.2:44359): tablet 3b7f9bbd946d4fcab6b9796a9ff0fdfc (table test-table [id=64edd173c7804868b45d00a2f27065e6]) successfully deleted
I20260928 05:18:00.341825 8104 tablet_service.cc:1467] Processing CreateTablet for tablet c1297cecd1ef43e386aeec2eb0fd3945 (DEFAULT_TABLE table=test-table [id=f60060ca1ab844e5b57742bf01617411]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:18:00.342190 8104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1297cecd1ef43e386aeec2eb0fd3945. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:18:00.342200 8382 tablet_service.cc:1467] Processing CreateTablet for tablet c1297cecd1ef43e386aeec2eb0fd3945 (DEFAULT_TABLE table=test-table [id=f60060ca1ab844e5b57742bf01617411]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:18:00.342581 8382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1297cecd1ef43e386aeec2eb0fd3945. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:18:00.348320 8477 tablet_bootstrap.cc:492] T c1297cecd1ef43e386aeec2eb0fd3945 P 0f89f414d76944cfb5a9505edaa66d35: Bootstrap starting.
I20260928 05:18:00.349946 8477 tablet_bootstrap.cc:654] T c1297cecd1ef43e386aeec2eb0fd3945 P 0f89f414d76944cfb5a9505edaa66d35: Neither blocks nor log segments found. Creating new log.
I20260928 05:18:00.350814 8478 tablet_bootstrap.cc:492] T c1297cecd1ef43e386aeec2eb0fd3945 P c35d4d4f8f3a49a09bfaec1f37147dd2: Bootstrap starting.
I20260928 05:18:00.352365 8478 tablet_bootstrap.cc:654] T c1297cecd1ef43e386aeec2eb0fd3945 P c35d4d4f8f3a49a09bfaec1f37147dd2: Neither blocks nor log segments found. Creating new log.
I20260928 05:18:00.361877 8477 tablet_bootstrap.cc:492] T c1297cecd1ef43e386aeec2eb0fd3945 P 0f89f414d76944cfb5a9505edaa66d35: No bootstrap required, opened a new log
I20260928 05:18:00.362037 8477 ts_tablet_manager.cc:1397] T c1297cecd1ef43e386aeec2eb0fd3945 P 0f89f414d76944cfb5a9505edaa66d35: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.009s
I20260928 05:18:00.362773 8477 raft_consensus.cc:348] T c1297cecd1ef43e386aeec2eb0fd3945 P 0f89f414d76944cfb5a9505edaa66d35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f89f414d76944cfb5a9505edaa66d35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42423 } } peers { permanent_uuid: "eeb06b1625b4449da5c75ca573e0f3e6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44359 } } peers { permanent_uuid: "c35d4d4f8f3a49a09bfaec1f37147dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35975 } }
I20260928 05:18:00.363025 8477 raft_consensus.cc:374] T c1297cecd1ef43e386aeec2eb0fd3945 P 0f89f414d76944cfb5a9505edaa66d35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:18:00.363111 8477 raft_consensus.cc:729] T c1297cecd1ef43e386aeec2eb0fd3945 P 0f89f414d76944cfb5a9505edaa66d35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f89f414d76944cfb5a9505edaa66d35, State: Initialized, Role: FOLLOWER
I20260928 05:18:00.363370 8477 consensus_queue.cc:260] T c1297cecd1ef43e386aeec2eb0fd3945 P 0f89f414d76944cfb5a9505edaa66d35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f89f414d76944cfb5a9505edaa66d35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42423 } } peers { permanent_uuid: "eeb06b1625b4449da5c75ca573e0f3e6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44359 } } peers { permanent_uuid: "c35d4d4f8f3a49a09bfaec1f37147dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35975 } }
I20260928 05:18:00.363883 8477 ts_tablet_manager.cc:1428] T c1297cecd1ef43e386aeec2eb0fd3945 P 0f89f414d76944cfb5a9505edaa66d35: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260928 05:18:00.364400 8476 tablet_bootstrap.cc:492] T c1297cecd1ef43e386aeec2eb0fd3945 P eeb06b1625b4449da5c75ca573e0f3e6: Bootstrap starting.
I20260928 05:18:00.365988 8476 tablet_bootstrap.cc:654] T c1297cecd1ef43e386aeec2eb0fd3945 P eeb06b1625b4449da5c75ca573e0f3e6: Neither blocks nor log segments found. Creating new log.
I20260928 05:18:00.368901 8476 tablet_bootstrap.cc:492] T c1297cecd1ef43e386aeec2eb0fd3945 P eeb06b1625b4449da5c75ca573e0f3e6: No bootstrap required, opened a new log
I20260928 05:18:00.369069 8476 ts_tablet_manager.cc:1397] T c1297cecd1ef43e386aeec2eb0fd3945 P eeb06b1625b4449da5c75ca573e0f3e6: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260928 05:18:00.369724 8476 raft_consensus.cc:348] T c1297cecd1ef43e386aeec2eb0fd3945 P eeb06b1625b4449da5c75ca573e0f3e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f89f414d76944cfb5a9505edaa66d35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42423 } } peers { permanent_uuid: "eeb06b1625b4449da5c75ca573e0f3e6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44359 } } peers { permanent_uuid: "c35d4d4f8f3a49a09bfaec1f37147dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35975 } }
I20260928 05:18:00.370129 8476 raft_consensus.cc:374] T c1297cecd1ef43e386aeec2eb0fd3945 P eeb06b1625b4449da5c75ca573e0f3e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:18:00.370276 8476 raft_consensus.cc:729] T c1297cecd1ef43e386aeec2eb0fd3945 P eeb06b1625b4449da5c75ca573e0f3e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeb06b1625b4449da5c75ca573e0f3e6, State: Initialized, Role: FOLLOWER
I20260928 05:18:00.370570 8476 consensus_queue.cc:260] T c1297cecd1ef43e386aeec2eb0fd3945 P eeb06b1625b4449da5c75ca573e0f3e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f89f414d76944cfb5a9505edaa66d35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42423 } } peers { permanent_uuid: "eeb06b1625b4449da5c75ca573e0f3e6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44359 } } peers { permanent_uuid: "c35d4d4f8f3a49a09bfaec1f37147dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35975 } }
I20260928 05:18:00.372888 8478 tablet_bootstrap.cc:492] T c1297cecd1ef43e386aeec2eb0fd3945 P c35d4d4f8f3a49a09bfaec1f37147dd2: No bootstrap required, opened a new log
I20260928 05:18:00.373055 8478 ts_tablet_manager.cc:1397] T c1297cecd1ef43e386aeec2eb0fd3945 P c35d4d4f8f3a49a09bfaec1f37147dd2: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.006s
I20260928 05:18:00.373804 8478 raft_consensus.cc:348] T c1297cecd1ef43e386aeec2eb0fd3945 P c35d4d4f8f3a49a09bfaec1f37147dd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f89f414d76944cfb5a9505edaa66d35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42423 } } peers { permanent_uuid: "eeb06b1625b4449da5c75ca573e0f3e6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44359 } } peers { permanent_uuid: "c35d4d4f8f3a49a09bfaec1f37147dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35975 } }
I20260928 05:18:00.374044 8478 raft_consensus.cc:374] T c1297cecd1ef43e386aeec2eb0fd3945 P c35d4d4f8f3a49a09bfaec1f37147dd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:18:00.374136 8478 raft_consensus.cc:729] T c1297cecd1ef43e386aeec2eb0fd3945 P c35d4d4f8f3a49a09bfaec1f37147dd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c35d4d4f8f3a49a09bfaec1f37147dd2, State: Initialized, Role: FOLLOWER
I20260928 05:18:00.374156 8476 ts_tablet_manager.cc:1428] T c1297cecd1ef43e386aeec2eb0fd3945 P eeb06b1625b4449da5c75ca573e0f3e6: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260928 05:18:00.374450 8478 consensus_queue.cc:260] T c1297cecd1ef43e386aeec2eb0fd3945 P c35d4d4f8f3a49a09bfaec1f37147dd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f89f414d76944cfb5a9505edaa66d35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42423 } } peers { permanent_uuid: "eeb06b1625b4449da5c75ca573e0f3e6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44359 } } peers { permanent_uuid: "c35d4d4f8f3a49a09bfaec1f37147dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35975 } }
I20260928 05:18:00.376475 8478 ts_tablet_manager.cc:1428] T c1297cecd1ef43e386aeec2eb0fd3945 P c35d4d4f8f3a49a09bfaec1f37147dd2: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260928 05:18:00.634047 8483 raft_consensus.cc:482] T c1297cecd1ef43e386aeec2eb0fd3945 P eeb06b1625b4449da5c75ca573e0f3e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:18:00.634274 8483 raft_consensus.cc:504] T c1297cecd1ef43e386aeec2eb0fd3945 P eeb06b1625b4449da5c75ca573e0f3e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f89f414d76944cfb5a9505edaa66d35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42423 } } peers { permanent_uuid: "eeb06b1625b4449da5c75ca573e0f3e6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44359 } } peers { permanent_uuid: "c35d4d4f8f3a49a09bfaec1f37147dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35975 } }
I20260928 05:18:00.635115 8483 leader_election.cc:288] T c1297cecd1ef43e386aeec2eb0fd3945 P eeb06b1625b4449da5c75ca573e0f3e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f89f414d76944cfb5a9505edaa66d35 (127.0.97.1:42423), c35d4d4f8f3a49a09bfaec1f37147dd2 (127.0.97.3:35975)
I20260928 05:18:00.635593 8123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1297cecd1ef43e386aeec2eb0fd3945" candidate_uuid: "eeb06b1625b4449da5c75ca573e0f3e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f89f414d76944cfb5a9505edaa66d35" is_pre_election: true
I20260928 05:18:00.635903 8123 raft_consensus.cc:2454] T c1297cecd1ef43e386aeec2eb0fd3945 P 0f89f414d76944cfb5a9505edaa66d35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeb06b1625b4449da5c75ca573e0f3e6 in term 0.
I20260928 05:18:00.635901 8407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1297cecd1ef43e386aeec2eb0fd3945" candidate_uuid: "eeb06b1625b4449da5c75ca573e0f3e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c35d4d4f8f3a49a09bfaec1f37147dd2" is_pre_election: true
I20260928 05:18:00.636188 8407 raft_consensus.cc:2454] T c1297cecd1ef43e386aeec2eb0fd3945 P c35d4d4f8f3a49a09bfaec1f37147dd2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeb06b1625b4449da5c75ca573e0f3e6 in term 0.
I20260928 05:18:00.636396 8197 leader_election.cc:302] T c1297cecd1ef43e386aeec2eb0fd3945 P eeb06b1625b4449da5c75ca573e0f3e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f89f414d76944cfb5a9505edaa66d35, eeb06b1625b4449da5c75ca573e0f3e6; no voters:
I20260928 05:18:00.636729 8483 raft_consensus.cc:2781] T c1297cecd1ef43e386aeec2eb0fd3945 P eeb06b1625b4449da5c75ca573e0f3e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 05:18:00.636844 8483 raft_consensus.cc:482] T c1297cecd1ef43e386aeec2eb0fd3945 P eeb06b1625b4449da5c75ca573e0f3e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 05:18:00.636951 8483 raft_consensus.cc:3037] T c1297cecd1ef43e386aeec2eb0fd3945 P eeb06b1625b4449da5c75ca573e0f3e6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:18:00.639894 8483 raft_consensus.cc:504] T c1297cecd1ef43e386aeec2eb0fd3945 P eeb06b1625b4449da5c75ca573e0f3e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f89f414d76944cfb5a9505edaa66d35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42423 } } peers { permanent_uuid: "eeb06b1625b4449da5c75ca573e0f3e6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44359 } } peers { permanent_uuid: "c35d4d4f8f3a49a09bfaec1f37147dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35975 } }
I20260928 05:18:00.641043 8123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1297cecd1ef43e386aeec2eb0fd3945" candidate_uuid: "eeb06b1625b4449da5c75ca573e0f3e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f89f414d76944cfb5a9505edaa66d35"
I20260928 05:18:00.641058 8407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1297cecd1ef43e386aeec2eb0fd3945" candidate_uuid: "eeb06b1625b4449da5c75ca573e0f3e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c35d4d4f8f3a49a09bfaec1f37147dd2"
I20260928 05:18:00.641237 8123 raft_consensus.cc:3037] T c1297cecd1ef43e386aeec2eb0fd3945 P 0f89f414d76944cfb5a9505edaa66d35 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:18:00.641279 8407 raft_consensus.cc:3037] T c1297cecd1ef43e386aeec2eb0fd3945 P c35d4d4f8f3a49a09bfaec1f37147dd2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:18:00.644536 8407 raft_consensus.cc:2454] T c1297cecd1ef43e386aeec2eb0fd3945 P c35d4d4f8f3a49a09bfaec1f37147dd2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeb06b1625b4449da5c75ca573e0f3e6 in term 1.
I20260928 05:18:00.644505 8123 raft_consensus.cc:2454] T c1297cecd1ef43e386aeec2eb0fd3945 P 0f89f414d76944cfb5a9505edaa66d35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeb06b1625b4449da5c75ca573e0f3e6 in term 1.
I20260928 05:18:00.645123 8197 leader_election.cc:302] T c1297cecd1ef43e386aeec2eb0fd3945 P eeb06b1625b4449da5c75ca573e0f3e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f89f414d76944cfb5a9505edaa66d35, eeb06b1625b4449da5c75ca573e0f3e6; no voters:
I20260928 05:18:00.646597 8483 leader_election.cc:288] T c1297cecd1ef43e386aeec2eb0fd3945 P eeb06b1625b4449da5c75ca573e0f3e6 [CANDIDATE]: Term 1 election: Requested vote from peers 0f89f414d76944cfb5a9505edaa66d35 (127.0.97.1:42423), c35d4d4f8f3a49a09bfaec1f37147dd2 (127.0.97.3:35975)
I20260928 05:18:00.647102 8483 raft_consensus.cc:2781] T c1297cecd1ef43e386aeec2eb0fd3945 P eeb06b1625b4449da5c75ca573e0f3e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:18:00.647239 8483 raft_consensus.cc:686] T c1297cecd1ef43e386aeec2eb0fd3945 P eeb06b1625b4449da5c75ca573e0f3e6 [term 1 LEADER]: Becoming Leader. State: Replica: eeb06b1625b4449da5c75ca573e0f3e6, State: Running, Role: LEADER
I20260928 05:18:00.647792 8483 consensus_queue.cc:237] T c1297cecd1ef43e386aeec2eb0fd3945 P eeb06b1625b4449da5c75ca573e0f3e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f89f414d76944cfb5a9505edaa66d35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42423 } } peers { permanent_uuid: "eeb06b1625b4449da5c75ca573e0f3e6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44359 } } peers { permanent_uuid: "c35d4d4f8f3a49a09bfaec1f37147dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35975 } }
I20260928 05:18:00.654323 7965 catalog_manager.cc:5495] T c1297cecd1ef43e386aeec2eb0fd3945 P eeb06b1625b4449da5c75ca573e0f3e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to eeb06b1625b4449da5c75ca573e0f3e6 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "eeb06b1625b4449da5c75ca573e0f3e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f89f414d76944cfb5a9505edaa66d35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eeb06b1625b4449da5c75ca573e0f3e6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c35d4d4f8f3a49a09bfaec1f37147dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35975 } health_report { overall_health: UNKNOWN } } }
I20260928 05:18:00.735730 8552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:00.731229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47462 (local address 127.0.97.2:44359)
0928 05:18:00.731382 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:00.731388 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:00.731465 (+ 77us) server_negotiation.cc:407] Connection header received
0928 05:18:00.731644 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:00.731649 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:00.731714 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:00.731783 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:18:00.732137 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:00.732915 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:00.734205 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:00.734621 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:00.734725 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:00.735129 (+ 404us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:18:00.735221 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:18:00.735291 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 05:18:00.735423 (+ 132us) server_negotiation.cc:299] Negotiation successful
0928 05:18:00.735494 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 05:18:00.749727 8570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:00.744131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35975 (local address 127.0.97.2:33347)
0928 05:18:00.744706 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0928 05:18:00.744720 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 05:18:00.744823 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:18:00.745159 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:18:00.745162 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:18:00.745171 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 05:18:00.745394 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:00.745400 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:00.746756 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:00.746759 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:00.747838 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:00.747849 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:00.748756 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:00.748760 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:00.748893 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:00.748945 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 05:18:00.749299 (+ 354us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 05:18:00.749312 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 05:18:00.749360 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 05:18:00.749449 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":296,"threads_started":1}
I20260928 05:18:00.749742 8473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:00.744157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33347 (local address 127.0.97.3:35975)
0928 05:18:00.744297 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:00.744301 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:00.744798 (+ 497us) server_negotiation.cc:407] Connection header received
0928 05:18:00.744949 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:00.744955 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:00.745057 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:00.745141 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:18:00.745522 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:00.746643 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:00.747967 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:00.748619 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:00.748729 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:00.749086 (+ 357us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:18:00.749211 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:18:00.749292 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 05:18:00.749380 (+ 88us) server_negotiation.cc:299] Negotiation successful
0928 05:18:00.749459 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 05:18:00.757470 8407 raft_consensus.cc:1261] T c1297cecd1ef43e386aeec2eb0fd3945 P c35d4d4f8f3a49a09bfaec1f37147dd2 [term 1 FOLLOWER]: Refusing update from remote peer eeb06b1625b4449da5c75ca573e0f3e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:18:00.757511 8123 raft_consensus.cc:1261] T c1297cecd1ef43e386aeec2eb0fd3945 P 0f89f414d76944cfb5a9505edaa66d35 [term 1 FOLLOWER]: Refusing update from remote peer eeb06b1625b4449da5c75ca573e0f3e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:18:00.758010 8483 consensus_queue.cc:1035] T c1297cecd1ef43e386aeec2eb0fd3945 P eeb06b1625b4449da5c75ca573e0f3e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f89f414d76944cfb5a9505edaa66d35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:18:00.758268 8564 consensus_queue.cc:1035] T c1297cecd1ef43e386aeec2eb0fd3945 P eeb06b1625b4449da5c75ca573e0f3e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c35d4d4f8f3a49a09bfaec1f37147dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:18:00.793090 8552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:00.777025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46207 (local address 127.0.97.2:44359)
0928 05:18:00.777167 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:00.777172 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:00.787692 (+ 10520us) server_negotiation.cc:407] Connection header received
0928 05:18:00.787987 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:00.787992 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:00.788080 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:00.788165 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:18:00.788650 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:00.789430 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:00.791374 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:00.791826 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:00.791916 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:00.792364 (+ 448us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:18:00.792483 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:18:00.792559 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 05:18:00.792712 (+ 153us) server_negotiation.cc:299] Negotiation successful
0928 05:18:00.792782 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 05:18:00.793149 8574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:00.776807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44359 (local address 127.0.97.3:46207)
0928 05:18:00.787600 (+ 10793us) negotiation.cc:107] Waiting for socket to connect
0928 05:18:00.787619 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 05:18:00.787792 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:18:00.788184 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:18:00.788189 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:18:00.788201 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 05:18:00.788507 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:00.788518 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:00.789542 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:00.789547 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:00.791250 (+ 1703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:00.791263 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:00.791987 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:00.791993 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:00.792160 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:00.792203 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 05:18:00.792584 (+ 381us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 05:18:00.792596 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 05:18:00.792668 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 05:18:00.792791 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10651,"thread_start_us":10787,"threads_started":1}
I20260928 05:18:00.825359 7965 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53274:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:18:00.825592 7965 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53274:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:18:00.828209 7965 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ca2877e519c94fd181176dfe63838bb3: Sending DeleteTablet for 3 replicas of tablet c1297cecd1ef43e386aeec2eb0fd3945
I20260928 05:18:00.828969 8104 tablet_service.cc:1514] Processing DeleteTablet for tablet c1297cecd1ef43e386aeec2eb0fd3945 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:18:00 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46714
I20260928 05:18:00.829051 8242 tablet_service.cc:1514] Processing DeleteTablet for tablet c1297cecd1ef43e386aeec2eb0fd3945 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:18:00 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47436
I20260928 05:18:00.829043 8382 tablet_service.cc:1514] Processing DeleteTablet for tablet c1297cecd1ef43e386aeec2eb0fd3945 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:18:00 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43410
I20260928 05:18:00.829233 8502 tablet_replica.cc:330] T c1297cecd1ef43e386aeec2eb0fd3945 P 0f89f414d76944cfb5a9505edaa66d35: stopping tablet replica
I20260928 05:18:00.829335 8501 tablet_replica.cc:330] T c1297cecd1ef43e386aeec2eb0fd3945 P eeb06b1625b4449da5c75ca573e0f3e6: stopping tablet replica
I20260928 05:18:00.829394 8502 raft_consensus.cc:2229] T c1297cecd1ef43e386aeec2eb0fd3945 P 0f89f414d76944cfb5a9505edaa66d35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:18:00.829557 8500 tablet_replica.cc:330] T c1297cecd1ef43e386aeec2eb0fd3945 P c35d4d4f8f3a49a09bfaec1f37147dd2: stopping tablet replica
I20260928 05:18:00.829564 8502 raft_consensus.cc:2258] T c1297cecd1ef43e386aeec2eb0fd3945 P 0f89f414d76944cfb5a9505edaa66d35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:18:00.829550 8501 raft_consensus.cc:2229] T c1297cecd1ef43e386aeec2eb0fd3945 P eeb06b1625b4449da5c75ca573e0f3e6 [term 1 LEADER]: Raft consensus shutting down.
I20260928 05:18:00.829738 8500 raft_consensus.cc:2229] T c1297cecd1ef43e386aeec2eb0fd3945 P c35d4d4f8f3a49a09bfaec1f37147dd2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:18:00.829906 8500 raft_consensus.cc:2258] T c1297cecd1ef43e386aeec2eb0fd3945 P c35d4d4f8f3a49a09bfaec1f37147dd2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:18:00.829906 8501 raft_consensus.cc:2258] T c1297cecd1ef43e386aeec2eb0fd3945 P eeb06b1625b4449da5c75ca573e0f3e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:18:00.830391 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 8038
I20260928 05:18:00.830717 8502 ts_tablet_manager.cc:1905] T c1297cecd1ef43e386aeec2eb0fd3945 P 0f89f414d76944cfb5a9505edaa66d35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:18:00.831324 8501 ts_tablet_manager.cc:1905] T c1297cecd1ef43e386aeec2eb0fd3945 P eeb06b1625b4449da5c75ca573e0f3e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:18:00.837123 8500 ts_tablet_manager.cc:1905] T c1297cecd1ef43e386aeec2eb0fd3945 P c35d4d4f8f3a49a09bfaec1f37147dd2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:18:00.837764 8502 ts_tablet_manager.cc:1918] T c1297cecd1ef43e386aeec2eb0fd3945 P 0f89f414d76944cfb5a9505edaa66d35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 05:18:00.837896 8502 log.cc:1196] T c1297cecd1ef43e386aeec2eb0fd3945 P 0f89f414d76944cfb5a9505edaa66d35: Deleting WAL directory at /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/minicluster-data/ts-0/wal/wals/c1297cecd1ef43e386aeec2eb0fd3945
I20260928 05:18:00.838392 8502 ts_tablet_manager.cc:1939] T c1297cecd1ef43e386aeec2eb0fd3945 P 0f89f414d76944cfb5a9505edaa66d35: Deleting consensus metadata
I20260928 05:18:00.839521 7954 catalog_manager.cc:4841] TS 0f89f414d76944cfb5a9505edaa66d35 (127.0.97.1:42423): tablet c1297cecd1ef43e386aeec2eb0fd3945 (table test-table [id=f60060ca1ab844e5b57742bf01617411]) successfully deleted
I20260928 05:18:00.842212 8501 ts_tablet_manager.cc:1918] T c1297cecd1ef43e386aeec2eb0fd3945 P eeb06b1625b4449da5c75ca573e0f3e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 05:18:00.842357 8501 log.cc:1196] T c1297cecd1ef43e386aeec2eb0fd3945 P eeb06b1625b4449da5c75ca573e0f3e6: Deleting WAL directory at /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/minicluster-data/ts-1/wal/wals/c1297cecd1ef43e386aeec2eb0fd3945
I20260928 05:18:00.842720 8501 ts_tablet_manager.cc:1939] T c1297cecd1ef43e386aeec2eb0fd3945 P eeb06b1625b4449da5c75ca573e0f3e6: Deleting consensus metadata
I20260928 05:18:00.843263 8500 ts_tablet_manager.cc:1918] T c1297cecd1ef43e386aeec2eb0fd3945 P c35d4d4f8f3a49a09bfaec1f37147dd2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 05:18:00.843430 8500 log.cc:1196] T c1297cecd1ef43e386aeec2eb0fd3945 P c35d4d4f8f3a49a09bfaec1f37147dd2: Deleting WAL directory at /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572622576160-388-0/minicluster-data/ts-2/wal/wals/c1297cecd1ef43e386aeec2eb0fd3945
I20260928 05:18:00.843535 7951 catalog_manager.cc:4841] TS eeb06b1625b4449da5c75ca573e0f3e6 (127.0.97.2:44359): tablet c1297cecd1ef43e386aeec2eb0fd3945 (table test-table [id=f60060ca1ab844e5b57742bf01617411]) successfully deleted
I20260928 05:18:00.843835 8500 ts_tablet_manager.cc:1939] T c1297cecd1ef43e386aeec2eb0fd3945 P c35d4d4f8f3a49a09bfaec1f37147dd2: Deleting consensus metadata
I20260928 05:18:00.844744 7953 catalog_manager.cc:4841] TS c35d4d4f8f3a49a09bfaec1f37147dd2 (127.0.97.3:35975): tablet c1297cecd1ef43e386aeec2eb0fd3945 (table test-table [id=f60060ca1ab844e5b57742bf01617411]) successfully deleted
Sep 28 05:18:00 dist-test-slave-whsn krb5kdc[7922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
I20260928 05:18:00.854254 8578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:00.831801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46726 (local address 127.0.97.1:42423)
0928 05:18:00.840118 (+ 8317us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:00.840124 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:00.840700 (+ 576us) server_negotiation.cc:407] Connection header received
0928 05:18:00.840928 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:00.840935 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:00.841026 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:00.841152 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:18:00.842648 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:00.843657 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:00.844981 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:00.845428 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:00.845528 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:00.849251 (+ 3723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:18:00.849265 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:18:00.849281 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:18:00.849310 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:18:00.851676 (+ 2366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:18:00.852277 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:18:00.852284 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:18:00.852286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:18:00.852406 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:18:00.852751 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:18:00.852755 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:18:00.852757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:18:00.853157 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:18:00.853436 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0928 05:18:00.853718 (+ 282us) server_negotiation.cc:299] Negotiation successful
0928 05:18:00.853938 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8185,"thread_start_us":8074,"threads_started":1}
I20260928 05:18:00.854890 8169 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:18:00.973793 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 8038
I20260928 05:18:00.992033 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 8178
Sep 28 05:18:00 dist-test-slave-whsn krb5kdc[7922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
I20260928 05:18:01.005414 8552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:00.992727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47466 (local address 127.0.97.2:44359)
0928 05:18:00.992946 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:00.992951 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:00.992969 (+ 18us) server_negotiation.cc:407] Connection header received
0928 05:18:00.993100 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:00.993104 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:00.993161 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:00.993237 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:18:00.994403 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:00.995085 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:00.996166 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:00.996557 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:00.996640 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:01.000238 (+ 3598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:18:01.000250 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:18:01.000264 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:18:01.000298 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:18:01.002892 (+ 2594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:18:01.003477 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:18:01.003484 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:18:01.003487 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:18:01.003617 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:18:01.003953 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:18:01.003957 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:18:01.003959 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:18:01.004302 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:18:01.004575 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0928 05:18:01.004932 (+ 357us) server_negotiation.cc:299] Negotiation successful
0928 05:18:01.005133 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 05:18:01.005966 8308 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:18:01.124562 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 8178
I20260928 05:18:01.141755 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 8317
Sep 28 05:18:01 dist-test-slave-whsn krb5kdc[7922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
I20260928 05:18:01.155298 8473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:01.142458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43436 (local address 127.0.97.3:35975)
0928 05:18:01.142660 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:01.142666 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:01.142687 (+ 21us) server_negotiation.cc:407] Connection header received
0928 05:18:01.142753 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:01.142758 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:01.142834 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:01.142936 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:18:01.144263 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:01.145024 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:01.146145 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:01.146533 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:01.146636 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:01.150703 (+ 4067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:18:01.150717 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:18:01.150733 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:18:01.150763 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:18:01.153208 (+ 2445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:18:01.153790 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:18:01.153797 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:18:01.153799 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:18:01.153926 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:18:01.154204 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:18:01.154207 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:18:01.154209 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:18:01.154526 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:18:01.154739 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0928 05:18:01.154931 (+ 192us) server_negotiation.cc:299] Negotiation successful
0928 05:18:01.155081 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 05:18:01.155869 8447 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:18:01.296032 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 8317
I20260928 05:18:01.315531 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 7935
I20260928 05:18:01.316632 8007 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:18:01.426549 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 7935
2026-09-28T05:18:01Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (5504 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790572622576160-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:18:01 dist-test-slave-whsn krb5kdc[8586](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:18:01 dist-test-slave-whsn krb5kdc[8586](info): set up 2 sockets
Sep 28 05:18:01 dist-test-slave-whsn krb5kdc[8586](info): commencing operation
krb5kdc: starting...
W20260928 05:18:02.417524 388 mini_kdc.cc:121] Time spent starting KDC: real 0.953s user 0.004s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790572622576160-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790572622576160-388-0/krb5kdc/test-user.keytab.
Sep 28 05:18:02 dist-test-slave-whsn krb5kdc[8586](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572682, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:18:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:18:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 05:18:02.604432 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790572622576160-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790572622576160-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790572622576160-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33733
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33733
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:18:02.905977 8599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:18:02.906255 8599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:18:02.906409 8599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:18:02.913511 8599 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:18:02.913614 8599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:18:02.913692 8599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:18:02.913779 8599 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:18:02.913847 8599 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:18:02.923585 8599 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790572622576160-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790572622576160-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33733
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790572622576160-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:18:02.925316 8599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:18:02.926656 8599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:18:02.937692 8607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:18:02.937763 8605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:18:02.937734 8604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:18:02.938704 8599 server_base.cc:1034] running on GCE node
I20260928 05:18:02.939419 8599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:18:02.940752 8599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:18:02.941987 8599 hybrid_clock.cc:648] HybridClock initialized: now 1790572682941933 us; error 38 us; skew 500 ppm
I20260928 05:18:02.942327 8599 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-28T05:18:03Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1584 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:18:03 dist-test-slave-whsn krb5kdc[8613](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:18:03 dist-test-slave-whsn krb5kdc[8613](info): set up 2 sockets
Sep 28 05:18:03 dist-test-slave-whsn krb5kdc[8613](info): commencing operation
krb5kdc: starting...
W20260928 05:18:04.009454 388 mini_kdc.cc:121] Time spent starting KDC: real 0.960s user 0.003s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/krb5kdc/test-user.keytab.
Sep 28 05:18:04 dist-test-slave-whsn krb5kdc[8613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572684, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:18:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:18:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 05:18:04.198055 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35487
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35487
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:18:04.493227 8626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:18:04.493531 8626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:18:04.493673 8626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:18:04.502041 8626 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:18:04.502190 8626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:18:04.502264 8626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:18:04.502341 8626 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:18:04.502400 8626 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:18:04.513867 8626 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35487
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:18:04.515110 8626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:18:04.516372 8626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:18:04.527472 8631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:18:04.527520 8632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:18:04.527558 8634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:18:04.528621 8626 server_base.cc:1034] running on GCE node
I20260928 05:18:04.529412 8626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:18:04.530726 8626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:18:04.531932 8626 hybrid_clock.cc:648] HybridClock initialized: now 1790572684531883 us; error 35 us; skew 500 ppm
I20260928 05:18:04.532238 8626 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:18:04 dist-test-slave-whsn krb5kdc[8613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572684, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:18:04.539076 8626 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 05:18:04.540490 8626 webserver.cc:460] Webserver started at http://127.0.97.62:37145/ using document root <none> and password file <none>
I20260928 05:18:04.541175 8626 fs_manager.cc:359] Metadata directory not provided
I20260928 05:18:04.541298 8626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:18:04.541591 8626 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:18:04.544436 8626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/master-0/data/instance:
uuid: "86a25c9408ab4490bcbd04194ed494c9"
format_stamp: "Formatted at 2026-09-28 05:18:04 on dist-test-slave-whsn"
server_key: "df154f373e3c323e59e2232a96a53734"
server_key_iv: "da747a892a91fc8789dc383e0248c7b6"
server_key_version: "encryptionkey@0"
I20260928 05:18:04.545269 8626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/master-0/wal/instance:
uuid: "86a25c9408ab4490bcbd04194ed494c9"
format_stamp: "Formatted at 2026-09-28 05:18:04 on dist-test-slave-whsn"
server_key: "df154f373e3c323e59e2232a96a53734"
server_key_iv: "da747a892a91fc8789dc383e0248c7b6"
server_key_version: "encryptionkey@0"
I20260928 05:18:04.550129 8626 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 05:18:04.553431 8641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:18:04.554641 8626 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 05:18:04.554795 8626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/master-0/wal
uuid: "86a25c9408ab4490bcbd04194ed494c9"
format_stamp: "Formatted at 2026-09-28 05:18:04 on dist-test-slave-whsn"
server_key: "df154f373e3c323e59e2232a96a53734"
server_key_iv: "da747a892a91fc8789dc383e0248c7b6"
server_key_version: "encryptionkey@0"
I20260928 05:18:04.554970 8626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:18:04.573621 8626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:18:04.577235 8626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:18:04.577466 8626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:18:04.593442 8626 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35487
I20260928 05:18:04.593484 8692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35487 every 8 connection(s)
I20260928 05:18:04.594652 8626 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:18:04.598004 8693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:18:04.603430 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 8626
I20260928 05:18:04.603642 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/master-0/wal/instance
I20260928 05:18:04.604413 388 external_mini_cluster.cc:1428] Setting key f53f651d1416181473c80900bc8f1d1e
I20260928 05:18:04.608177 8693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86a25c9408ab4490bcbd04194ed494c9: Bootstrap starting.
I20260928 05:18:04.611471 8693 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 86a25c9408ab4490bcbd04194ed494c9: Neither blocks nor log segments found. Creating new log.
I20260928 05:18:04.612390 8693 log.cc:824] T 00000000000000000000000000000000 P 86a25c9408ab4490bcbd04194ed494c9: Log is configured to *not* fsync() on all Append() calls
I20260928 05:18:04.615043 8693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86a25c9408ab4490bcbd04194ed494c9: No bootstrap required, opened a new log
Sep 28 05:18:04 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572684, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:18:04.622592 8693 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86a25c9408ab4490bcbd04194ed494c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86a25c9408ab4490bcbd04194ed494c9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35487 } }
I20260928 05:18:04.622916 8693 raft_consensus.cc:374] T 00000000000000000000000000000000 P 86a25c9408ab4490bcbd04194ed494c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:18:04.623005 8693 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86a25c9408ab4490bcbd04194ed494c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86a25c9408ab4490bcbd04194ed494c9, State: Initialized, Role: FOLLOWER
I20260928 05:18:04.623541 8693 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86a25c9408ab4490bcbd04194ed494c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86a25c9408ab4490bcbd04194ed494c9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35487 } }
I20260928 05:18:04.623780 8693 raft_consensus.cc:388] T 00000000000000000000000000000000 P 86a25c9408ab4490bcbd04194ed494c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:18:04.623903 8693 raft_consensus.cc:482] T 00000000000000000000000000000000 P 86a25c9408ab4490bcbd04194ed494c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:18:04.624104 8693 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86a25c9408ab4490bcbd04194ed494c9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:18:04.625216 8696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:04.606367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52792 (local address 127.0.97.62:35487)
0928 05:18:04.607065 (+ 698us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:04.607076 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:04.607114 (+ 38us) server_negotiation.cc:407] Connection header received
0928 05:18:04.608733 (+ 1619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:04.608754 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:04.609145 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:04.609455 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:18:04.611077 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:04.612340 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:04.613795 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:04.614410 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:04.614538 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:04.618480 (+ 3942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:18:04.618491 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:18:04.618503 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:18:04.618535 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:18:04.621463 (+ 2928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:18:04.622080 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:18:04.622086 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:18:04.622092 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:18:04.622239 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:18:04.622576 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:18:04.622582 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:18:04.622585 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:18:04.623117 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:18:04.623390 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0928 05:18:04.623784 (+ 394us) server_negotiation.cc:299] Negotiation successful
0928 05:18:04.624155 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":333,"threads_started":1}
I20260928 05:18:04.626664 8693 raft_consensus.cc:504] T 00000000000000000000000000000000 P 86a25c9408ab4490bcbd04194ed494c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86a25c9408ab4490bcbd04194ed494c9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35487 } }
I20260928 05:18:04.627235 8693 leader_election.cc:302] T 00000000000000000000000000000000 P 86a25c9408ab4490bcbd04194ed494c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86a25c9408ab4490bcbd04194ed494c9; no voters:
I20260928 05:18:04.627835 8693 leader_election.cc:288] T 00000000000000000000000000000000 P 86a25c9408ab4490bcbd04194ed494c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:18:04.627978 8698 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 86a25c9408ab4490bcbd04194ed494c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:18:04.628614 8698 raft_consensus.cc:686] T 00000000000000000000000000000000 P 86a25c9408ab4490bcbd04194ed494c9 [term 1 LEADER]: Becoming Leader. State: Replica: 86a25c9408ab4490bcbd04194ed494c9, State: Running, Role: LEADER
I20260928 05:18:04.629118 8698 consensus_queue.cc:237] T 00000000000000000000000000000000 P 86a25c9408ab4490bcbd04194ed494c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86a25c9408ab4490bcbd04194ed494c9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35487 } }
I20260928 05:18:04.629760 8693 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86a25c9408ab4490bcbd04194ed494c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:18:04.631582 8700 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86a25c9408ab4490bcbd04194ed494c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86a25c9408ab4490bcbd04194ed494c9. Latest consensus state: current_term: 1 leader_uuid: "86a25c9408ab4490bcbd04194ed494c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86a25c9408ab4490bcbd04194ed494c9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35487 } } }
I20260928 05:18:04.632053 8700 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86a25c9408ab4490bcbd04194ed494c9 [sys.catalog]: This master's current role is: LEADER
I20260928 05:18:04.631968 8699 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86a25c9408ab4490bcbd04194ed494c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "86a25c9408ab4490bcbd04194ed494c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86a25c9408ab4490bcbd04194ed494c9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35487 } } }
I20260928 05:18:04.632930 8699 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86a25c9408ab4490bcbd04194ed494c9 [sys.catalog]: This master's current role is: LEADER
I20260928 05:18:04.633850 8708 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:18:04.638377 8708 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:18:04.646047 8708 catalog_manager.cc:1347] Generated new cluster ID: 432d4097165e4b9180e1174a561aaa21
I20260928 05:18:04.646132 8708 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:18:04.660692 8708 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:18:04.662400 8708 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:18:04.671083 8708 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 86a25c9408ab4490bcbd04194ed494c9: Generated new TSK 0
I20260928 05:18:04.672019 8708 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260928 05:18:04.718046 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35487
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:18:05.018684 8719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:18:05.018954 8719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:18:05.019090 8719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:18:05.026090 8719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:18:05.026201 8719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:18:05.026347 8719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 05:18:05.035898 8719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35487
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:18:05.037135 8719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:18:05.038339 8719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:18:05.050179 8724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:18:05.050302 8725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:18:05.050606 8727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:18:05.051575 8719 server_base.cc:1034] running on GCE node
I20260928 05:18:05.052135 8719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:18:05.053027 8719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:18:05.054261 8719 hybrid_clock.cc:648] HybridClock initialized: now 1790572685054202 us; error 49 us; skew 500 ppm
I20260928 05:18:05.054534 8719 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:18:05 dist-test-slave-whsn krb5kdc[8613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572685, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:18:05.057554 8719 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260928 05:18:05.058985 8719 webserver.cc:460] Webserver started at http://127.0.97.1:43111/ using document root <none> and password file <none>
I20260928 05:18:05.059656 8719 fs_manager.cc:359] Metadata directory not provided
I20260928 05:18:05.059754 8719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:18:05.060009 8719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:18:05.062752 8719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/ts-0/data/instance:
uuid: "82385f49333749e6a4b65e294a997078"
format_stamp: "Formatted at 2026-09-28 05:18:05 on dist-test-slave-whsn"
server_key: "95bab071b929c9474ed51d1b1adf9168"
server_key_iv: "75a64643672615570752c0930488b483"
server_key_version: "encryptionkey@0"
I20260928 05:18:05.063426 8719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/ts-0/wal/instance:
uuid: "82385f49333749e6a4b65e294a997078"
format_stamp: "Formatted at 2026-09-28 05:18:05 on dist-test-slave-whsn"
server_key: "95bab071b929c9474ed51d1b1adf9168"
server_key_iv: "75a64643672615570752c0930488b483"
server_key_version: "encryptionkey@0"
I20260928 05:18:05.068077 8719 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 05:18:05.071233 8734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:18:05.072436 8719 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 05:18:05.072568 8719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/ts-0/wal
uuid: "82385f49333749e6a4b65e294a997078"
format_stamp: "Formatted at 2026-09-28 05:18:05 on dist-test-slave-whsn"
server_key: "95bab071b929c9474ed51d1b1adf9168"
server_key_iv: "75a64643672615570752c0930488b483"
server_key_version: "encryptionkey@0"
I20260928 05:18:05.072728 8719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:18:05.109988 8719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:18:05.113935 8719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:18:05.114221 8719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:18:05.115137 8719 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:18:05.116472 8719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:18:05.116555 8719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:18:05.116680 8719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:18:05.116739 8719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:18:05.147697 8719 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:32827
I20260928 05:18:05.147753 8846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:32827 every 8 connection(s)
I20260928 05:18:05.149040 8719 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/ts-0/data/info.pb
I20260928 05:18:05.156803 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 8719
I20260928 05:18:05.157017 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/ts-0/wal/instance
I20260928 05:18:05.157581 388 external_mini_cluster.cc:1428] Setting key bf909a5b9303e36d64ff373130f5bb42
Sep 28 05:18:05 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572685, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:18:05.167373 8849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:05.151403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59759 (local address 127.0.97.62:35487)
0928 05:18:05.151817 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:05.151822 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:05.152685 (+ 863us) server_negotiation.cc:407] Connection header received
0928 05:18:05.154458 (+ 1773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:05.154471 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:05.154552 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:05.154643 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:18:05.156041 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:05.156754 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:05.158508 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:05.158970 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:05.159073 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:05.162377 (+ 3304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:18:05.162382 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:18:05.162385 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:18:05.162415 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:18:05.164598 (+ 2183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:18:05.165327 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:18:05.165331 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:18:05.165333 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:18:05.165438 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:18:05.165825 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:18:05.165829 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:18:05.165833 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:18:05.166085 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:18:05.166210 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 05:18:05.166830 (+ 620us) server_negotiation.cc:299] Negotiation successful
0928 05:18:05.167034 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":221,"threads_started":1}
I20260928 05:18:05.168280 8850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:05.151829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35487 (local address 127.0.97.1:59759)
0928 05:18:05.152523 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0928 05:18:05.152560 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 05:18:05.154206 (+ 1646us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:18:05.154876 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:18:05.154884 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:18:05.155354 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:18:05.155882 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:05.155899 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:05.156917 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:05.156922 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:05.158378 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:05.158393 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:05.159164 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:05.159172 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:05.159397 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:05.160028 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:18:05.160049 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:18:05.162181 (+ 2132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:18:05.164728 (+ 2547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:18:05.164737 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:18:05.164753 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:18:05.165213 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:18:05.165541 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:18:05.165546 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:18:05.165550 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:18:05.165719 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:18:05.166198 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:18:05.166206 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:18:05.166506 (+ 300us) client_negotiation.cc:769] Sending connection context
0928 05:18:05.166726 (+ 220us) client_negotiation.cc:240] Negotiation successful
0928 05:18:05.167202 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":288,"threads_started":1}
I20260928 05:18:05.169680 8847 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35487
I20260928 05:18:05.169991 8847 heartbeater.cc:464] Registering TS with master...
I20260928 05:18:05.170733 8847 heartbeater.cc:511] Master 127.0.97.62:35487 requested a full tablet report, sending...
I20260928 05:18:05.172901 8658 ts_manager.cc:194] Registered new tserver with Master: 82385f49333749e6a4b65e294a997078 (127.0.97.1:32827)
I20260928 05:18:05.174857 8658 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:59759
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260928 05:18:05.201835 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35487
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:18:05.498306 8853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:18:05.498543 8853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:18:05.498641 8853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:18:05.505358 8853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:18:05.505445 8853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:18:05.505550 8853 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 05:18:05.514420 8853 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35487
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:18:05.515489 8853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:18:05.516647 8853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:18:05.528406 8858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:18:05.528434 8859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:18:05.528486 8861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:18:05.529356 8853 server_base.cc:1034] running on GCE node
I20260928 05:18:05.529935 8853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:18:05.530812 8853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:18:05.532020 8853 hybrid_clock.cc:648] HybridClock initialized: now 1790572685531962 us; error 51 us; skew 500 ppm
I20260928 05:18:05.532308 8853 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:18:05 dist-test-slave-whsn krb5kdc[8613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572685, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:18:05.535244 8853 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260928 05:18:05.536474 8853 webserver.cc:460] Webserver started at http://127.0.97.2:45055/ using document root <none> and password file <none>
I20260928 05:18:05.537107 8853 fs_manager.cc:359] Metadata directory not provided
I20260928 05:18:05.537202 8853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:18:05.537446 8853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:18:05.540001 8853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/ts-1/data/instance:
uuid: "69d4b2fec1184aed9ccf942ee1c42866"
format_stamp: "Formatted at 2026-09-28 05:18:05 on dist-test-slave-whsn"
server_key: "3e976431d2f15b60a928183dfd203890"
server_key_iv: "1edfd383ec327d22e6b2378d8118734c"
server_key_version: "encryptionkey@0"
I20260928 05:18:05.540664 8853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/ts-1/wal/instance:
uuid: "69d4b2fec1184aed9ccf942ee1c42866"
format_stamp: "Formatted at 2026-09-28 05:18:05 on dist-test-slave-whsn"
server_key: "3e976431d2f15b60a928183dfd203890"
server_key_iv: "1edfd383ec327d22e6b2378d8118734c"
server_key_version: "encryptionkey@0"
I20260928 05:18:05.545154 8853 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260928 05:18:05.548166 8868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:18:05.549418 8853 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 05:18:05.549535 8853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/ts-1/wal
uuid: "69d4b2fec1184aed9ccf942ee1c42866"
format_stamp: "Formatted at 2026-09-28 05:18:05 on dist-test-slave-whsn"
server_key: "3e976431d2f15b60a928183dfd203890"
server_key_iv: "1edfd383ec327d22e6b2378d8118734c"
server_key_version: "encryptionkey@0"
I20260928 05:18:05.549682 8853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:18:05.573025 8853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:18:05.576737 8853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:18:05.577033 8853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:18:05.577937 8853 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:18:05.579118 8853 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:18:05.579190 8853 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:18:05.579299 8853 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:18:05.579349 8853 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:18:05.608538 8853 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36179
I20260928 05:18:05.608636 8980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36179 every 8 connection(s)
I20260928 05:18:05.609737 8853 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/ts-1/data/info.pb
I20260928 05:18:05.618204 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 8853
I20260928 05:18:05.618362 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/ts-1/wal/instance
I20260928 05:18:05.618726 388 external_mini_cluster.cc:1428] Setting key 14bd4e1bf8db714a83023217d70a12ba
Sep 28 05:18:05 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572685, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:18:05.629120 8849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:05.612075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:32777 (local address 127.0.97.62:35487)
0928 05:18:05.612230 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:05.612234 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:05.613347 (+ 1113us) server_negotiation.cc:407] Connection header received
0928 05:18:05.615186 (+ 1839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:05.615193 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:05.615274 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:05.615369 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:18:05.616831 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:05.617595 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:05.619205 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:05.619763 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:05.619881 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:05.623229 (+ 3348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:18:05.623234 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:18:05.623238 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:18:05.623271 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:18:05.626151 (+ 2880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:18:05.626940 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:18:05.626946 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:18:05.626950 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:18:05.627121 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:18:05.627479 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:18:05.627484 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:18:05.627488 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:18:05.627823 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:18:05.627968 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 05:18:05.628539 (+ 571us) server_negotiation.cc:299] Negotiation successful
0928 05:18:05.628757 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 05:18:05.629818 8983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:05.612478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35487 (local address 127.0.97.2:32777)
0928 05:18:05.613197 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0928 05:18:05.613233 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 05:18:05.614926 (+ 1693us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:18:05.615617 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:18:05.615626 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:18:05.616113 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:18:05.616660 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:05.616678 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:05.617746 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:05.617749 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:05.619056 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:05.619068 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:05.619998 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:05.620006 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:05.620295 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:05.620944 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:18:05.620967 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:18:05.623021 (+ 2054us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:18:05.626295 (+ 3274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:18:05.626304 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:18:05.626319 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:18:05.626821 (+ 502us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:18:05.627224 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:18:05.627227 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:18:05.627229 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:18:05.627387 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:18:05.627937 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:18:05.627947 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:18:05.628211 (+ 264us) client_negotiation.cc:769] Sending connection context
0928 05:18:05.628416 (+ 205us) client_negotiation.cc:240] Negotiation successful
0928 05:18:05.628752 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":258,"threads_started":1}
I20260928 05:18:05.631203 8981 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35487
I20260928 05:18:05.631523 8981 heartbeater.cc:464] Registering TS with master...
I20260928 05:18:05.632319 8981 heartbeater.cc:511] Master 127.0.97.62:35487 requested a full tablet report, sending...
I20260928 05:18:05.633905 8658 ts_manager.cc:194] Registered new tserver with Master: 69d4b2fec1184aed9ccf942ee1c42866 (127.0.97.2:36179)
I20260928 05:18:05.634757 8658 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:32777
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260928 05:18:05.661824 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35487
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:18:05.956815 8986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:18:05.957095 8986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:18:05.957222 8986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:18:05.964277 8986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:18:05.964370 8986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:18:05.964519 8986 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 05:18:05.973711 8986 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35487
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:18:05.974882 8986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:18:05.976167 8986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:18:05.988310 8992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:18:05.988344 8994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:18:05.988365 8991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:18:05.990116 8986 server_base.cc:1034] running on GCE node
I20260928 05:18:05.990653 8986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:18:05.991495 8986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:18:05.992686 8986 hybrid_clock.cc:648] HybridClock initialized: now 1790572685992643 us; error 36 us; skew 500 ppm
I20260928 05:18:05.993032 8986 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:18:05 dist-test-slave-whsn krb5kdc[8613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572685, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:18:05.995848 8986 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260928 05:18:05.997268 8986 webserver.cc:460] Webserver started at http://127.0.97.3:37915/ using document root <none> and password file <none>
I20260928 05:18:05.997910 8986 fs_manager.cc:359] Metadata directory not provided
I20260928 05:18:05.998031 8986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:18:05.998298 8986 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:18:06.001031 8986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/ts-2/data/instance:
uuid: "8260f4839eba442685cea6dae998f460"
format_stamp: "Formatted at 2026-09-28 05:18:05 on dist-test-slave-whsn"
server_key: "7952bafa60871cfb6059010081f19cf6"
server_key_iv: "a4a6d3ec870bade3c145c2846a7ca094"
server_key_version: "encryptionkey@0"
I20260928 05:18:06.001739 8986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/ts-2/wal/instance:
uuid: "8260f4839eba442685cea6dae998f460"
format_stamp: "Formatted at 2026-09-28 05:18:05 on dist-test-slave-whsn"
server_key: "7952bafa60871cfb6059010081f19cf6"
server_key_iv: "a4a6d3ec870bade3c145c2846a7ca094"
server_key_version: "encryptionkey@0"
I20260928 05:18:06.006412 8986 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260928 05:18:06.009555 9001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:18:06.010774 8986 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:18:06.010905 8986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/ts-2/wal
uuid: "8260f4839eba442685cea6dae998f460"
format_stamp: "Formatted at 2026-09-28 05:18:05 on dist-test-slave-whsn"
server_key: "7952bafa60871cfb6059010081f19cf6"
server_key_iv: "a4a6d3ec870bade3c145c2846a7ca094"
server_key_version: "encryptionkey@0"
I20260928 05:18:06.011065 8986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:18:06.034622 8986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:18:06.038187 8986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:18:06.038422 8986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:18:06.039249 8986 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:18:06.040398 8986 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:18:06.040482 8986 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:18:06.040597 8986 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:18:06.040647 8986 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:18:06.069228 8986 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39055
I20260928 05:18:06.069293 9113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39055 every 8 connection(s)
I20260928 05:18:06.070397 8986 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/ts-2/data/info.pb
I20260928 05:18:06.078002 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 8986
I20260928 05:18:06.078181 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/ts-2/wal/instance
I20260928 05:18:06.078647 388 external_mini_cluster.cc:1428] Setting key 537890d04aad36d14a732b2aabdbb6dc
Sep 28 05:18:06 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572685, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:18:06.088563 8849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:06.072629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49627 (local address 127.0.97.62:35487)
0928 05:18:06.072811 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:06.072817 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:06.073968 (+ 1151us) server_negotiation.cc:407] Connection header received
0928 05:18:06.075759 (+ 1791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:06.075764 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:06.075839 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:06.075974 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:18:06.077385 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:06.078373 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:06.080036 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:06.080573 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:06.080664 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:06.083930 (+ 3266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:18:06.083935 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:18:06.083938 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:18:06.083976 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:18:06.086189 (+ 2213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:18:06.086799 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:18:06.086802 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:18:06.086804 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:18:06.086905 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:18:06.087284 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:18:06.087287 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:18:06.087289 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:18:06.087529 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:18:06.087629 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 05:18:06.088192 (+ 563us) server_negotiation.cc:299] Negotiation successful
0928 05:18:06.088336 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 05:18:06.089432 9116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:06.073171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35487 (local address 127.0.97.3:49627)
0928 05:18:06.073832 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0928 05:18:06.073867 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 05:18:06.075511 (+ 1644us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:18:06.076168 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:18:06.076176 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:18:06.076640 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:18:06.077242 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:06.077259 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:06.078507 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:06.078510 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:06.079910 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:06.079921 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:06.080779 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:06.080786 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:06.081099 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:06.081762 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:18:06.081783 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:18:06.083755 (+ 1972us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:18:06.086307 (+ 2552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:18:06.086313 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:18:06.086324 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:18:06.086712 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:18:06.087026 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:18:06.087030 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:18:06.087033 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:18:06.087202 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:18:06.087617 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:18:06.087622 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:18:06.087883 (+ 261us) client_negotiation.cc:769] Sending connection context
0928 05:18:06.088091 (+ 208us) client_negotiation.cc:240] Negotiation successful
0928 05:18:06.088433 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":260,"threads_started":1}
I20260928 05:18:06.090704 9114 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35487
I20260928 05:18:06.090971 9114 heartbeater.cc:464] Registering TS with master...
I20260928 05:18:06.091601 9114 heartbeater.cc:511] Master 127.0.97.62:35487 requested a full tablet report, sending...
I20260928 05:18:06.092904 8658 ts_manager.cc:194] Registered new tserver with Master: 8260f4839eba442685cea6dae998f460 (127.0.97.3:39055)
I20260928 05:18:06.093854 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 05:18:06.093858 8658 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:49627
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/krb5kdc/oryx_127.0.97.1.keytab.
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260928 05:18:06.178122 8847 heartbeater.cc:503] Master 127.0.97.62:35487 was elected leader, sending a full tablet report...
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260928 05:18:06.199350 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 8719
Sep 28 05:18:06 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572684, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260928 05:18:06.213238 9124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:06.200303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53554 (local address 127.0.97.1:32827)
0928 05:18:06.200932 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:06.200939 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:06.200964 (+ 25us) server_negotiation.cc:407] Connection header received
0928 05:18:06.201031 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:06.201037 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:06.201237 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:06.201362 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:18:06.202510 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:06.203462 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:06.204551 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:06.205133 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:06.205219 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:06.208694 (+ 3475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:18:06.208705 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:18:06.208719 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:18:06.208748 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:18:06.211156 (+ 2408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:18:06.211698 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:18:06.211703 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:18:06.211705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:18:06.211813 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:18:06.212070 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:18:06.212073 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:18:06.212075 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:18:06.212360 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:18:06.212565 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0928 05:18:06.212757 (+ 192us) server_negotiation.cc:299] Negotiation successful
0928 05:18:06.212964 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":310,"threads_started":1}
I20260928 05:18:06.214531 8842 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:18:06.294443 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 8719
I20260928 05:18:06.306833 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 8853
Sep 28 05:18:06 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572684, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260928 05:18:06.321026 9126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:06.307650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38128 (local address 127.0.97.2:36179)
0928 05:18:06.308141 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:06.308147 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:06.308170 (+ 23us) server_negotiation.cc:407] Connection header received
0928 05:18:06.308225 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:06.308231 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:06.308435 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:06.308553 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:18:06.309685 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:06.310646 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:06.311787 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:06.312295 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:06.312382 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:06.316114 (+ 3732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:18:06.316123 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:18:06.316135 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:18:06.316172 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:18:06.318537 (+ 2365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:18:06.319114 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:18:06.319120 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:18:06.319122 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:18:06.319230 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:18:06.319534 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:18:06.319539 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:18:06.319542 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:18:06.319838 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:18:06.320106 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0928 05:18:06.320370 (+ 264us) server_negotiation.cc:299] Negotiation successful
0928 05:18:06.320628 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":234,"threads_started":1}
I20260928 05:18:06.322171 8976 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:18:06.401072 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 8853
I20260928 05:18:06.413408 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 8986
Sep 28 05:18:06 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572684, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260928 05:18:06.428508 9128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:06.414239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55116 (local address 127.0.97.3:39055)
0928 05:18:06.414822 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:06.414829 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:06.414858 (+ 29us) server_negotiation.cc:407] Connection header received
0928 05:18:06.414918 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:06.414925 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:06.415138 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:06.415266 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:18:06.416519 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:06.417568 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:06.418802 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:06.419326 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:06.419416 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:06.423647 (+ 4231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:18:06.423657 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:18:06.423671 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:18:06.423705 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:18:06.426216 (+ 2511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:18:06.426796 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:18:06.426801 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:18:06.426804 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:18:06.426912 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:18:06.427207 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:18:06.427209 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:18:06.427211 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:18:06.427508 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:18:06.427724 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0928 05:18:06.427923 (+ 199us) server_negotiation.cc:299] Negotiation successful
0928 05:18:06.428172 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":294,"threads_started":1}
I20260928 05:18:06.429862 9109 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:18:06.511457 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 8986
I20260928 05:18:06.523502 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 8626
I20260928 05:18:06.524611 8688 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:18:06.622044 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 8626
I20260928 05:18:06.638055 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35487
--webserver_interface=127.0.97.62
--webserver_port=37145
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35487
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:18:06.934000 9131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:18:06.934291 9131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:18:06.934414 9131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:18:06.941262 9131 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:18:06.941363 9131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:18:06.941433 9131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:18:06.941502 9131 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:18:06.941563 9131 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:18:06.950758 9131 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35487
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=37145
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:18:06.951920 9131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:18:06.953137 9131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:18:06.963722 9139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:18:06.963732 9137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:18:06.963939 9136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:18:06.964291 9131 server_base.cc:1034] running on GCE node
I20260928 05:18:06.965052 9131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:18:06.966295 9131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:18:06.967478 9131 hybrid_clock.cc:648] HybridClock initialized: now 1790572686967431 us; error 35 us; skew 500 ppm
I20260928 05:18:06.967784 9131 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:18:06 dist-test-slave-whsn krb5kdc[8613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572686, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:18:06.970642 9131 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 05:18:06.971931 9131 webserver.cc:460] Webserver started at http://127.0.97.62:37145/ using document root <none> and password file <none>
I20260928 05:18:06.972543 9131 fs_manager.cc:359] Metadata directory not provided
I20260928 05:18:06.972671 9131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:18:06.977672 9131 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 05:18:06.980113 9146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:18:06.981338 9131 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 05:18:06.981480 9131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/master-0/wal
uuid: "86a25c9408ab4490bcbd04194ed494c9"
format_stamp: "Formatted at 2026-09-28 05:18:04 on dist-test-slave-whsn"
server_key: "df154f373e3c323e59e2232a96a53734"
server_key_iv: "da747a892a91fc8789dc383e0248c7b6"
server_key_version: "encryptionkey@0"
I20260928 05:18:06.982333 9131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:18:06.995762 9131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:18:06.999742 9131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:18:06.999996 9131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:18:07.016424 9131 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35487
I20260928 05:18:07.016443 9197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35487 every 8 connection(s)
I20260928 05:18:07.017920 9131 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:18:07.024363 9198 sys_catalog.cc:263] Verifying existing consensus state
I20260928 05:18:07.024603 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 9131
I20260928 05:18:07.025365 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:32827
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:35487
--webserver_port=43111
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.1
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/krb5kdc/oryx_127.0.97.1.keytab with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 05:18:07.027576 9198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86a25c9408ab4490bcbd04194ed494c9: Bootstrap starting.
I20260928 05:18:07.051067 9198 log.cc:824] T 00000000000000000000000000000000 P 86a25c9408ab4490bcbd04194ed494c9: Log is configured to *not* fsync() on all Append() calls
I20260928 05:18:07.061605 9198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86a25c9408ab4490bcbd04194ed494c9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 05:18:07.062171 9198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86a25c9408ab4490bcbd04194ed494c9: Bootstrap complete.
I20260928 05:18:07.074400 9198 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86a25c9408ab4490bcbd04194ed494c9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86a25c9408ab4490bcbd04194ed494c9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35487 } }
I20260928 05:18:07.075495 9198 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86a25c9408ab4490bcbd04194ed494c9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86a25c9408ab4490bcbd04194ed494c9, State: Initialized, Role: FOLLOWER
I20260928 05:18:07.076174 9198 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86a25c9408ab4490bcbd04194ed494c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86a25c9408ab4490bcbd04194ed494c9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35487 } }
I20260928 05:18:07.076452 9198 raft_consensus.cc:388] T 00000000000000000000000000000000 P 86a25c9408ab4490bcbd04194ed494c9 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:18:07.076566 9198 raft_consensus.cc:482] T 00000000000000000000000000000000 P 86a25c9408ab4490bcbd04194ed494c9 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:18:07.076735 9198 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86a25c9408ab4490bcbd04194ed494c9 [term 1 FOLLOWER]: Advancing to term 2
I20260928 05:18:07.081899 9198 raft_consensus.cc:504] T 00000000000000000000000000000000 P 86a25c9408ab4490bcbd04194ed494c9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86a25c9408ab4490bcbd04194ed494c9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35487 } }
I20260928 05:18:07.082436 9198 leader_election.cc:302] T 00000000000000000000000000000000 P 86a25c9408ab4490bcbd04194ed494c9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86a25c9408ab4490bcbd04194ed494c9; no voters:
I20260928 05:18:07.082999 9198 leader_election.cc:288] T 00000000000000000000000000000000 P 86a25c9408ab4490bcbd04194ed494c9 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 05:18:07.083112 9202 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 86a25c9408ab4490bcbd04194ed494c9 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 05:18:07.084107 9202 raft_consensus.cc:686] T 00000000000000000000000000000000 P 86a25c9408ab4490bcbd04194ed494c9 [term 2 LEADER]: Becoming Leader. State: Replica: 86a25c9408ab4490bcbd04194ed494c9, State: Running, Role: LEADER
I20260928 05:18:07.084595 9202 consensus_queue.cc:237] T 00000000000000000000000000000000 P 86a25c9408ab4490bcbd04194ed494c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86a25c9408ab4490bcbd04194ed494c9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35487 } }
I20260928 05:18:07.085151 9198 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86a25c9408ab4490bcbd04194ed494c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:18:07.086763 9203 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86a25c9408ab4490bcbd04194ed494c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "86a25c9408ab4490bcbd04194ed494c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86a25c9408ab4490bcbd04194ed494c9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35487 } } }
I20260928 05:18:07.087090 9203 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86a25c9408ab4490bcbd04194ed494c9 [sys.catalog]: This master's current role is: LEADER
I20260928 05:18:07.087824 9204 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86a25c9408ab4490bcbd04194ed494c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86a25c9408ab4490bcbd04194ed494c9. Latest consensus state: current_term: 2 leader_uuid: "86a25c9408ab4490bcbd04194ed494c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86a25c9408ab4490bcbd04194ed494c9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35487 } } }
I20260928 05:18:07.088183 9204 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86a25c9408ab4490bcbd04194ed494c9 [sys.catalog]: This master's current role is: LEADER
I20260928 05:18:07.098955 9208 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:18:07.104076 9208 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:18:07.105399 9208 catalog_manager.cc:1259] Loaded cluster ID: 432d4097165e4b9180e1174a561aaa21
I20260928 05:18:07.105535 9208 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:18:07.109267 9208 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:18:07.110589 9208 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 86a25c9408ab4490bcbd04194ed494c9: Loaded TSK: 0
I20260928 05:18:07.112140 9208 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 05:18:07.352572 9200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:18:07.352804 9200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:18:07.352942 9200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:18:07.359841 9200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:18:07.359936 9200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:18:07.360075 9200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 05:18:07.369064 9200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:32827
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=43111
--tserver_master_addrs=127.0.97.62:35487
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:18:07.370185 9200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:18:07.371546 9200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:18:07.383503 9225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:18:07.383711 9224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:18:07.384020 9227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:18:07.384505 9200 server_base.cc:1034] running on GCE node
I20260928 05:18:07.385156 9200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:18:07.386111 9200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:18:07.387312 9200 hybrid_clock.cc:648] HybridClock initialized: now 1790572687387258 us; error 50 us; skew 500 ppm
I20260928 05:18:07.387594 9200 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:18:07 dist-test-slave-whsn krb5kdc[8613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572687, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:18:07.390708 9200 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260928 05:18:07.392091 9200 webserver.cc:460] Webserver started at http://127.0.97.1:43111/ using document root <none> and password file <none>
I20260928 05:18:07.392705 9200 fs_manager.cc:359] Metadata directory not provided
I20260928 05:18:07.392825 9200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:18:07.397835 9200 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260928 05:18:07.400375 9234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:18:07.401643 9200 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 05:18:07.401770 9200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/ts-0/wal
uuid: "82385f49333749e6a4b65e294a997078"
format_stamp: "Formatted at 2026-09-28 05:18:05 on dist-test-slave-whsn"
server_key: "95bab071b929c9474ed51d1b1adf9168"
server_key_iv: "75a64643672615570752c0930488b483"
server_key_version: "encryptionkey@0"
I20260928 05:18:07.402698 9200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:18:07.423311 9200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:18:07.426929 9200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:18:07.427158 9200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:18:07.428078 9200 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:18:07.429374 9200 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:18:07.429471 9200 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:18:07.429566 9200 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:18:07.429636 9200 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:18:07.457500 9200 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:32827
I20260928 05:18:07.457561 9346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:32827 every 8 connection(s)
I20260928 05:18:07.458755 9200 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/ts-0/data/info.pb
I20260928 05:18:07.466995 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 9200
I20260928 05:18:07.467523 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:36179
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:35487
--webserver_port=45055
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.2
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/krb5kdc/oryx_127.0.97.2.keytab with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 05:18:07 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:18:07 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:18:07.476344 9350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:07.461552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35487 (local address 127.0.97.1:60837)
0928 05:18:07.462384 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0928 05:18:07.462437 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 05:18:07.464121 (+ 1684us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:18:07.465522 (+ 1401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:18:07.465532 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:18:07.466119 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:18:07.466702 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:07.466721 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:07.468473 (+ 1752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:07.468478 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:07.469735 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:07.469748 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:07.470773 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:07.470781 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:07.471053 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:07.471675 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:18:07.471704 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:18:07.475113 (+ 3409us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":374,"threads_started":1}
W20260928 05:18:07.476426 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:18:07.461482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60837 (local address 127.0.97.62:35487)
0928 05:18:07.462354 (+ 872us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:07.462370 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:07.462550 (+ 180us) server_negotiation.cc:407] Connection header received
0928 05:18:07.464655 (+ 2105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:07.464669 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:07.465037 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:07.465357 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:18:07.466919 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:07.468315 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:07.469875 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:07.470536 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:07.470719 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:07.475471 (+ 4752us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:60837: BlockingRecv error: recv got EOF from 127.0.97.1:60837 (error 108)
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":346,"threads_started":1}
W20260928 05:18:07.477145 9347 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:35487 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:35487: Client connection negotiation failed: client connection to 127.0.97.62:35487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Sep 28 05:18:07 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:18:07 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:18:07.485081 9350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:07.477700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35487 (local address 127.0.97.1:53617)
0928 05:18:07.477831 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 05:18:07.477850 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 05:18:07.477940 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:18:07.478197 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:18:07.478199 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:18:07.478499 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:18:07.478644 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:07.478652 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:07.479943 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:07.479947 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:07.480823 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:07.480834 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:07.481720 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:07.481725 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:07.481851 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:07.482398 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:18:07.482415 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:18:07.484809 (+ 2394us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 05:18:07.485277 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:18:07.477775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53617 (local address 127.0.97.62:35487)
0928 05:18:07.477915 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:07.477920 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:07.477946 (+ 26us) server_negotiation.cc:407] Connection header received
0928 05:18:07.478023 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:07.478028 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:07.478090 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:07.478190 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:18:07.478766 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:07.479775 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:07.480988 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:07.481565 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:07.481691 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:07.485063 (+ 3372us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:53617: BlockingRecv error: recv got EOF from 127.0.97.1:53617 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 28 05:18:07 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:18:07 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:18:07.493196 9350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:07.485737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35487 (local address 127.0.97.1:36299)
0928 05:18:07.485865 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 05:18:07.485881 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 05:18:07.485982 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:18:07.486496 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:18:07.486500 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:18:07.486735 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:18:07.486857 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:07.486864 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:07.488119 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:07.488122 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:07.489054 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:07.489064 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:07.489877 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:07.489882 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:07.490014 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:07.490480 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:18:07.490496 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:18:07.492936 (+ 2440us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":29}
W20260928 05:18:07.493229 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:18:07.486078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36299 (local address 127.0.97.62:35487)
0928 05:18:07.486225 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:07.486231 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:07.486247 (+ 16us) server_negotiation.cc:407] Connection header received
0928 05:18:07.486288 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:07.486293 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:07.486367 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:07.486458 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:18:07.486966 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:07.487988 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:07.489161 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:07.489759 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:07.489867 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:07.493063 (+ 3196us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:36299: BlockingRecv error: recv got EOF from 127.0.97.1:36299 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260928 05:18:07.493525 9347 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 05:18:07.764298 9351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:18:07.764557 9351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:18:07.764710 9351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:18:07.771495 9351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:18:07.771598 9351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:18:07.771737 9351 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 05:18:07.780728 9351 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:36179
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=45055
--tserver_master_addrs=127.0.97.62:35487
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:18:07.781884 9351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:18:07.783067 9351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:18:07.794754 9356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:18:07.794706 9357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:18:07.794791 9359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:18:07.795545 9351 server_base.cc:1034] running on GCE node
I20260928 05:18:07.796059 9351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:18:07.796918 9351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:18:07.798098 9351 hybrid_clock.cc:648] HybridClock initialized: now 1790572687798052 us; error 42 us; skew 500 ppm
I20260928 05:18:07.798375 9351 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:18:07 dist-test-slave-whsn krb5kdc[8613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572687, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:18:07.801196 9351 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260928 05:18:07.802450 9351 webserver.cc:460] Webserver started at http://127.0.97.2:45055/ using document root <none> and password file <none>
I20260928 05:18:07.803033 9351 fs_manager.cc:359] Metadata directory not provided
I20260928 05:18:07.803141 9351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:18:07.808135 9351 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260928 05:18:07.810573 9366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:18:07.811771 9351 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260928 05:18:07.811903 9351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/ts-1/wal
uuid: "69d4b2fec1184aed9ccf942ee1c42866"
format_stamp: "Formatted at 2026-09-28 05:18:05 on dist-test-slave-whsn"
server_key: "3e976431d2f15b60a928183dfd203890"
server_key_iv: "1edfd383ec327d22e6b2378d8118734c"
server_key_version: "encryptionkey@0"
I20260928 05:18:07.812745 9351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:18:07.824327 9351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:18:07.827613 9351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:18:07.827827 9351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:18:07.828634 9351 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:18:07.829813 9351 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:18:07.829901 9351 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:18:07.829993 9351 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:18:07.830062 9351 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:18:07.858500 9351 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36179
I20260928 05:18:07.858563 9478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36179 every 8 connection(s)
I20260928 05:18:07.859869 9351 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/ts-1/data/info.pb
I20260928 05:18:07.864455 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 9351
I20260928 05:18:07.865175 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:39055
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:35487
--webserver_port=37915
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.3
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/krb5kdc/oryx_127.0.97.3.keytab with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 05:18:07 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:18:07 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 05:18:07.875988 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:18:07.862191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46767 (local address 127.0.97.62:35487)
0928 05:18:07.862408 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:07.862415 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:07.863504 (+ 1089us) server_negotiation.cc:407] Connection header received
0928 05:18:07.865771 (+ 2267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:07.865776 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:07.865849 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:07.865956 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:18:07.867759 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:07.868501 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:07.870514 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:07.870959 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:07.871065 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:07.875802 (+ 4737us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:46767: BlockingRecv error: recv got EOF from 127.0.97.2:46767 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 05:18:07.876780 9481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:07.862696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35487 (local address 127.0.97.2:46767)
0928 05:18:07.863345 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0928 05:18:07.863386 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 05:18:07.865489 (+ 2103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:18:07.866202 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:18:07.866213 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:18:07.866867 (+ 654us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:18:07.867581 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:07.867603 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:07.868652 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:07.868657 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:07.870373 (+ 1716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:07.870386 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:07.871159 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:07.871166 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:07.871416 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:07.872071 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:18:07.872092 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:18:07.875663 (+ 3571us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":261,"threads_started":1}
W20260928 05:18:07.877504 9479 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:35487 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:35487: Client connection negotiation failed: client connection to 127.0.97.62:35487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Sep 28 05:18:07 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:18:07 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:18:07.884812 9481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:07.878026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35487 (local address 127.0.97.2:54619)
0928 05:18:07.878160 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 05:18:07.878178 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 05:18:07.878266 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:18:07.878553 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:18:07.878558 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:18:07.878857 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:18:07.878976 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:07.878983 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:07.879867 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:07.879872 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:07.880723 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:07.880733 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:07.881387 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:07.881393 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:07.881551 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:07.882028 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:18:07.882044 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:18:07.884528 (+ 2484us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":41}
W20260928 05:18:07.885129 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:18:07.878163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54619 (local address 127.0.97.62:35487)
0928 05:18:07.878333 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:07.878338 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:07.878353 (+ 15us) server_negotiation.cc:407] Connection header received
0928 05:18:07.878392 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:07.878395 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:07.878446 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:07.878521 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:18:07.879098 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:07.879746 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:07.880843 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:07.881256 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:07.881367 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:07.884764 (+ 3397us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:54619: BlockingRecv error: recv got EOF from 127.0.97.2:54619 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 28 05:18:07 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:18:07 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:18:07.892803 9481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:07.885552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35487 (local address 127.0.97.2:47735)
0928 05:18:07.885699 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 05:18:07.885716 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 05:18:07.885820 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:18:07.886138 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:18:07.886143 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:18:07.886503 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:18:07.886658 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:07.886666 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:07.887962 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:07.887965 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:07.888819 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:07.888829 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:07.889696 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:07.889700 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:07.889829 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:07.890315 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:18:07.890330 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:18:07.892622 (+ 2292us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":38}
W20260928 05:18:07.893133 9479 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 05:18:07.893447 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:18:07.885683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47735 (local address 127.0.97.62:35487)
0928 05:18:07.885880 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:07.885886 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:07.885903 (+ 17us) server_negotiation.cc:407] Connection header received
0928 05:18:07.885947 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:07.885952 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:07.886015 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:07.886123 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:18:07.886764 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:07.887847 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:07.888965 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:07.889585 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:07.889697 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:07.892836 (+ 3139us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:47735: BlockingRecv error: recv got EOF from 127.0.97.2:47735 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260928 05:18:08.175493 9482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:18:08.175737 9482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:18:08.175829 9482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:18:08.182641 9482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:18:08.182731 9482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:18:08.182842 9482 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 05:18:08.191987 9482 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:39055
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=37915
--tserver_master_addrs=127.0.97.62:35487
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:18:08.193181 9482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:18:08.194351 9482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:18:08.205889 9490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:18:08.205958 9487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:18:08.205931 9488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:18:08.206810 9482 server_base.cc:1034] running on GCE node
I20260928 05:18:08.207346 9482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:18:08.208168 9482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:18:08.209344 9482 hybrid_clock.cc:648] HybridClock initialized: now 1790572688209290 us; error 53 us; skew 500 ppm
I20260928 05:18:08.209610 9482 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:18:08 dist-test-slave-whsn krb5kdc[8613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572688, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:18:08.212285 9482 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260928 05:18:08.213613 9482 webserver.cc:460] Webserver started at http://127.0.97.3:37915/ using document root <none> and password file <none>
I20260928 05:18:08.214208 9482 fs_manager.cc:359] Metadata directory not provided
I20260928 05:18:08.214323 9482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:18:08.219183 9482 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 05:18:08.221617 9497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:18:08.222857 9482 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:18:08.222998 9482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/ts-2/wal
uuid: "8260f4839eba442685cea6dae998f460"
format_stamp: "Formatted at 2026-09-28 05:18:05 on dist-test-slave-whsn"
server_key: "7952bafa60871cfb6059010081f19cf6"
server_key_iv: "a4a6d3ec870bade3c145c2846a7ca094"
server_key_version: "encryptionkey@0"
I20260928 05:18:08.223865 9482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:18:08.237524 9482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:18:08.240993 9482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:18:08.241216 9482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:18:08.242051 9482 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:18:08.243225 9482 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:18:08.243315 9482 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:18:08.243408 9482 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:18:08.243484 9482 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:18:08.271924 9482 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39055
I20260928 05:18:08.271974 9609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39055 every 8 connection(s)
I20260928 05:18:08.273306 9482 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572622576160-388-0/minicluster-data/ts-2/data/info.pb
I20260928 05:18:08.275311 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 9482
Sep 28 05:18:08 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:18:08 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 05:18:08.289712 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:18:08.275722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51099 (local address 127.0.97.62:35487)
0928 05:18:08.275934 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:08.275940 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:08.277315 (+ 1375us) server_negotiation.cc:407] Connection header received
0928 05:18:08.279705 (+ 2390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:08.279710 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:08.279793 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:08.279877 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:18:08.281304 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:08.282043 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:08.283537 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:08.284217 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:08.284315 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:08.289453 (+ 5138us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:51099: BlockingRecv error: recv got EOF from 127.0.97.3:51099 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 05:18:08.290411 9614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:08.276226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35487 (local address 127.0.97.3:51099)
0928 05:18:08.277140 (+ 914us) negotiation.cc:107] Waiting for socket to connect
0928 05:18:08.277194 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 05:18:08.279479 (+ 2285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:18:08.280136 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:18:08.280143 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:18:08.280603 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:18:08.281161 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:08.281178 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:08.282165 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:08.282169 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:08.283403 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:08.283427 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:08.284403 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:08.284411 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:08.284704 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:08.285409 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:18:08.285447 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:18:08.289264 (+ 3817us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":363,"threads_started":1}
W20260928 05:18:08.291059 9610 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:35487 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:35487: Client connection negotiation failed: client connection to 127.0.97.62:35487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
I20260928 05:18:08.293042 9613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:08.276171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52806 (local address 127.0.97.62:35487)
0928 05:18:08.279394 (+ 3223us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:08.279400 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:08.279427 (+ 27us) server_negotiation.cc:407] Connection header received
0928 05:18:08.279486 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:08.279491 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:08.279566 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:08.280933 (+ 1367us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:18:08.281167 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:08.281865 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:08.283038 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:08.283467 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:08.283574 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:08.286982 (+ 3408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:18:08.286993 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:18:08.287011 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:18:08.287056 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:18:08.289806 (+ 2750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:18:08.290568 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:18:08.290578 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:18:08.290586 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:18:08.290773 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:18:08.291226 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:18:08.291231 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:18:08.291235 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:18:08.291782 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:18:08.292069 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0928 05:18:08.292400 (+ 331us) server_negotiation.cc:299] Negotiation successful
0928 05:18:08.292659 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3124,"thread_start_us":3012,"threads_started":1}
Sep 28 05:18:08 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:18:08 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:18:08.298954 9614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:08.291571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35487 (local address 127.0.97.3:52687)
0928 05:18:08.291698 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 05:18:08.291714 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 05:18:08.291797 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:18:08.292139 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:18:08.292143 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:18:08.292509 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:18:08.292659 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:08.292670 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:08.293770 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:08.293775 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:08.295005 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:08.295018 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:08.295801 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:08.295807 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:08.295968 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:08.296420 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:18:08.296445 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:18:08.298777 (+ 2332us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 05:18:08.299196 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:18:08.291628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52687 (local address 127.0.97.62:35487)
0928 05:18:08.291758 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:08.291762 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:08.291778 (+ 16us) server_negotiation.cc:407] Connection header received
0928 05:18:08.291910 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:08.291915 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:08.292014 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:08.292100 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:18:08.292783 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:08.293656 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:08.295126 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:08.295675 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:08.295778 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:08.298981 (+ 3203us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:52687: BlockingRecv error: recv got EOF from 127.0.97.3:52687 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 28 05:18:08 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:18:08 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:18:08.305816 9614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:08.299463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35487 (local address 127.0.97.3:43997)
0928 05:18:08.299601 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 05:18:08.299615 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 05:18:08.299700 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:18:08.299949 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:18:08.299953 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:18:08.300247 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:18:08.300396 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:08.300407 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:08.301329 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:08.301334 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:08.302197 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:08.302207 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:08.302783 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:08.302789 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:08.302944 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:08.303337 (+ 393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:18:08.303351 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:18:08.305630 (+ 2279us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":42}
W20260928 05:18:08.306025 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:18:08.299546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43997 (local address 127.0.97.62:35487)
0928 05:18:08.299740 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:08.299745 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:08.299760 (+ 15us) server_negotiation.cc:407] Connection header received
0928 05:18:08.299799 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:08.299802 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:08.299846 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:08.299916 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:18:08.300519 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:08.301214 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:08.302314 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:08.302667 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:08.302764 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:08.305770 (+ 3006us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:43997: BlockingRecv error: recv got EOF from 127.0.97.3:43997 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
W20260928 05:18:08.306066 9610 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 28 05:18:08 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:18:08 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:18:08.504951 9615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:08.494142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35487 (local address 127.0.97.1:47981)
0928 05:18:08.496756 (+ 2614us) negotiation.cc:107] Waiting for socket to connect
0928 05:18:08.496776 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 05:18:08.496934 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:18:08.497247 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:18:08.497252 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:18:08.497772 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:18:08.498011 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:08.498024 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:08.499340 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:08.499345 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:08.500420 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:08.500432 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:08.501222 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:08.501228 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:08.501366 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:08.501869 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:18:08.501911 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:18:08.504565 (+ 2654us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":2507,"thread_start_us":2379,"threads_started":1}
W20260928 05:18:08.505028 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:18:08.494228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47981 (local address 127.0.97.62:35487)
0928 05:18:08.494438 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:08.494445 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:08.496846 (+ 2401us) server_negotiation.cc:407] Connection header received
0928 05:18:08.497066 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:08.497069 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:08.497129 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:08.497202 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:18:08.498177 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:08.499203 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:08.500544 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:08.501053 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:08.501166 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:08.504779 (+ 3613us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:47981: BlockingRecv error: recv got EOF from 127.0.97.1:47981 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 28 05:18:08 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:18:08 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:18:08.901386 9616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:08.893697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35487 (local address 127.0.97.2:42997)
0928 05:18:08.894148 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0928 05:18:08.894167 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 05:18:08.894278 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:18:08.894544 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:18:08.894547 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:18:08.894975 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:18:08.895162 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:08.895171 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:08.896042 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:08.896045 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:08.897000 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:08.897010 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:08.897606 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:08.897610 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:08.897721 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:08.898246 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:18:08.898264 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:18:08.901176 (+ 2912us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":241,"threads_started":1}
W20260928 05:18:08.901554 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:18:08.893750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42997 (local address 127.0.97.62:35487)
0928 05:18:08.893997 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:08.894003 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:08.894257 (+ 254us) server_negotiation.cc:407] Connection header received
0928 05:18:08.894369 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:08.894373 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:08.894436 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:08.894514 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:18:08.895261 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:08.895924 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:08.897090 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:08.897479 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:08.897561 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:08.901344 (+ 3783us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:42997: BlockingRecv error: recv got EOF from 127.0.97.2:42997 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 28 05:18:09 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:18:09 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:18:09.314296 9617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:09.306610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35487 (local address 127.0.97.3:55547)
0928 05:18:09.307092 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0928 05:18:09.307113 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 05:18:09.307233 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:18:09.307494 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:18:09.307497 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:18:09.307985 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:18:09.308192 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:09.308201 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:09.309228 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:09.309232 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:09.310226 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:09.310239 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:09.310911 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:09.310917 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:09.311046 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:09.311563 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:18:09.311579 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:18:09.314083 (+ 2504us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":293,"threads_started":1}
W20260928 05:18:09.314441 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:18:09.306661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55547 (local address 127.0.97.62:35487)
0928 05:18:09.306814 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:09.306820 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:09.307185 (+ 365us) server_negotiation.cc:407] Connection header received
0928 05:18:09.307318 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:09.307322 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:09.307382 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:09.307466 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:18:09.308325 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:09.309083 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:09.310360 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:09.310734 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:09.310823 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:09.314243 (+ 3420us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:55547: BlockingRecv error: recv got EOF from 127.0.97.3:55547 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 28 05:18:09 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:18:09 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:18:09.513576 9618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:09.505959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35487 (local address 127.0.97.1:53635)
0928 05:18:09.506420 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0928 05:18:09.506445 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 05:18:09.506551 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:18:09.506803 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:18:09.506806 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:18:09.507187 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:18:09.507344 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:09.507353 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:09.508300 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:09.508303 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:09.509222 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:09.509234 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:09.509875 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:09.509880 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:09.510008 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:09.510555 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:18:09.510572 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:18:09.513339 (+ 2767us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":255,"threads_started":1}
W20260928 05:18:09.513700 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:18:09.506016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53635 (local address 127.0.97.62:35487)
0928 05:18:09.506205 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:09.506212 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:09.506509 (+ 297us) server_negotiation.cc:407] Connection header received
0928 05:18:09.506636 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:09.506640 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:09.506700 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:09.506775 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:18:09.507455 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:09.508189 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:09.509339 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:09.509716 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:09.509811 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:09.513496 (+ 3685us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:53635: BlockingRecv error: recv got EOF from 127.0.97.1:53635 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 28 05:18:09 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:18:09 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:18:09.910279 9619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:09.902163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35487 (local address 127.0.97.2:53775)
0928 05:18:09.902627 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0928 05:18:09.902642 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 05:18:09.902769 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:18:09.903109 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:18:09.903114 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:18:09.903495 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:18:09.903640 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:09.903652 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:09.904979 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:09.904985 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:09.905912 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:09.905923 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:09.906707 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:09.906712 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:09.906835 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:09.907428 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:18:09.907447 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:18:09.910053 (+ 2606us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":254,"threads_started":1}
W20260928 05:18:09.910398 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:18:09.902235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53775 (local address 127.0.97.62:35487)
0928 05:18:09.902400 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:09.902406 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:09.902721 (+ 315us) server_negotiation.cc:407] Connection header received
0928 05:18:09.902883 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:09.902888 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:09.902947 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:09.903048 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:18:09.903786 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:09.904833 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:09.906038 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:09.906559 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:09.906648 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:09.910217 (+ 3569us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:53775: BlockingRecv error: recv got EOF from 127.0.97.2:53775 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 28 05:18:10 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:18:10 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:18:10.322738 9620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:10.315054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35487 (local address 127.0.97.3:34125)
0928 05:18:10.315514 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0928 05:18:10.315530 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 05:18:10.315657 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:18:10.315975 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:18:10.315980 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:18:10.316387 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:18:10.316548 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:10.316558 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:10.317556 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:10.317562 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:10.318432 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:10.318452 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:10.319105 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:10.319112 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:10.319258 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:10.319779 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:18:10.319796 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:18:10.322429 (+ 2633us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":226,"threads_started":1}
W20260928 05:18:10.322863 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:18:10.315232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34125 (local address 127.0.97.62:35487)
0928 05:18:10.315445 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:10.315451 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:10.315600 (+ 149us) server_negotiation.cc:407] Connection header received
0928 05:18:10.315762 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:10.315765 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:10.315843 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:10.315918 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:18:10.316681 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:10.317431 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:10.318569 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:10.318972 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:10.319071 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:10.322612 (+ 3541us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:34125: BlockingRecv error: recv got EOF from 127.0.97.3:34125 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 28 05:18:10 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:18:10 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:18:10.522190 9621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:10.514335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35487 (local address 127.0.97.1:53979)
0928 05:18:10.514772 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0928 05:18:10.514788 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 05:18:10.514904 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:18:10.515189 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:18:10.515191 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:18:10.515538 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:18:10.515694 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:10.515704 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:10.516779 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:10.516783 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:10.517788 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:10.517802 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:10.518572 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:10.518578 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:10.518731 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:10.519276 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:18:10.519294 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:18:10.521959 (+ 2665us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":238,"threads_started":1}
W20260928 05:18:10.522370 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:18:10.514419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53979 (local address 127.0.97.62:35487)
0928 05:18:10.514579 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:10.514583 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:10.514849 (+ 266us) server_negotiation.cc:407] Connection header received
0928 05:18:10.515012 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:10.515015 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:10.515080 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:10.515156 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:18:10.515831 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:10.516631 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:10.517922 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:10.518435 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:10.518544 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:10.522145 (+ 3601us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:53979: BlockingRecv error: recv got EOF from 127.0.97.1:53979 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 28 05:18:10 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:18:10 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:18:10.918943 9622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:10.911129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35487 (local address 127.0.97.2:58525)
0928 05:18:10.911591 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0928 05:18:10.911611 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 05:18:10.911764 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:18:10.912108 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:18:10.912113 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:18:10.912454 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:18:10.912618 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:10.912627 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:10.913671 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:10.913677 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:10.914550 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:10.914561 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:10.915200 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:10.915205 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:10.915378 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:10.915945 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:18:10.915965 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:18:10.918705 (+ 2740us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":278,"threads_started":1}
W20260928 05:18:10.919126 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:18:10.911239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58525 (local address 127.0.97.62:35487)
0928 05:18:10.911410 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:10.911416 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:10.911688 (+ 272us) server_negotiation.cc:407] Connection header received
0928 05:18:10.911877 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:10.911883 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:10.911976 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:10.912052 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:18:10.912753 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:10.913540 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:10.914673 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:10.915078 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:10.915177 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:10.918901 (+ 3724us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:58525: BlockingRecv error: recv got EOF from 127.0.97.2:58525 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 28 05:18:11 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:18:11 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:18:11.331585 9623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:11.323822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35487 (local address 127.0.97.3:50169)
0928 05:18:11.324301 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0928 05:18:11.324317 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 05:18:11.324447 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:18:11.324768 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:18:11.324773 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:18:11.325152 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:18:11.325318 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:11.325327 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:11.326307 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:11.326313 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:11.327352 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:11.327365 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:11.328001 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:11.328007 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:11.328163 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:11.328712 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:18:11.328728 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:18:11.331268 (+ 2540us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":265,"threads_started":1}
W20260928 05:18:11.331684 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:18:11.323950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50169 (local address 127.0.97.62:35487)
0928 05:18:11.324129 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:11.324135 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:11.324394 (+ 259us) server_negotiation.cc:407] Connection header received
0928 05:18:11.324555 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:11.324559 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:11.324643 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:11.324721 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:18:11.325465 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:11.326198 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:11.327482 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:11.327868 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:11.327969 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:11.331482 (+ 3513us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:50169: BlockingRecv error: recv got EOF from 127.0.97.3:50169 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 28 05:18:11 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:18:11 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:18:11.530776 9624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:11.522994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35487 (local address 127.0.97.1:33191)
0928 05:18:11.523471 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0928 05:18:11.523486 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 05:18:11.523603 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:18:11.523858 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:18:11.523861 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:18:11.524285 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:18:11.524472 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:11.524498 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:11.525654 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:11.525658 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:11.526626 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:11.526637 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:11.527284 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:11.527290 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:11.527460 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:11.527996 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:18:11.528012 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:18:11.530483 (+ 2471us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":257,"threads_started":1}
W20260928 05:18:11.530889 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:18:11.523077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33191 (local address 127.0.97.62:35487)
0928 05:18:11.523250 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:11.523255 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:11.523553 (+ 298us) server_negotiation.cc:407] Connection header received
0928 05:18:11.523688 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:11.523692 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:11.523753 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:11.523829 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:18:11.524655 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:11.525533 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:11.526763 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:11.527157 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:11.527265 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:11.530671 (+ 3406us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:33191: BlockingRecv error: recv got EOF from 127.0.97.1:33191 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 28 05:18:11 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:18:11 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:18:11.927248 9625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:11.919737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35487 (local address 127.0.97.2:36911)
0928 05:18:11.920174 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0928 05:18:11.920189 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 05:18:11.920324 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:18:11.920637 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:18:11.920642 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:18:11.921030 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:18:11.921174 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:11.921182 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:11.922118 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:11.922123 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:11.923069 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:11.923080 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:11.923715 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:11.923720 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:11.923890 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:11.924410 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:18:11.924427 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:18:11.927031 (+ 2604us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":253,"threads_started":1}
W20260928 05:18:11.927373 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:18:11.919800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36911 (local address 127.0.97.62:35487)
0928 05:18:11.919963 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:11.919977 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:11.920268 (+ 291us) server_negotiation.cc:407] Connection header received
0928 05:18:11.920438 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:11.920443 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:11.920507 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:11.920591 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:18:11.921304 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:11.922003 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:11.923199 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:11.923593 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:11.923692 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:11.927182 (+ 3490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:36911: BlockingRecv error: recv got EOF from 127.0.97.2:36911 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 28 05:18:12 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:18:12 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:18:12.340145 9626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:12.332502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35487 (local address 127.0.97.3:48631)
0928 05:18:12.332985 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0928 05:18:12.333001 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 05:18:12.333106 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:18:12.333425 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:18:12.333429 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:18:12.333852 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:18:12.334036 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:12.334051 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:12.335158 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:12.335162 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:12.336086 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:12.336097 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:12.336664 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:12.336668 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:12.336790 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:12.337318 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:18:12.337335 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:18:12.339844 (+ 2509us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":225,"threads_started":1}
W20260928 05:18:12.340293 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:18:12.332591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48631 (local address 127.0.97.62:35487)
0928 05:18:12.332753 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:12.332758 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:12.333077 (+ 319us) server_negotiation.cc:407] Connection header received
0928 05:18:12.333207 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:12.333211 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:12.333274 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:12.333372 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:18:12.334173 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:12.335033 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:12.336188 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:12.336565 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:12.336649 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:12.340075 (+ 3426us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:48631: BlockingRecv error: recv got EOF from 127.0.97.3:48631 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 28 05:18:12 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:18:12 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:18:12.538836 9627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:12.531634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35487 (local address 127.0.97.1:48677)
0928 05:18:12.532127 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0928 05:18:12.532143 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 05:18:12.532242 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:18:12.532481 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:18:12.532483 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:18:12.532823 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:18:12.533040 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:12.533049 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:12.533959 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:12.533963 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:12.534866 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:12.534877 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:12.535489 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:12.535493 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:12.535602 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:12.536097 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:18:12.536114 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:18:12.538609 (+ 2495us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":238,"threads_started":1}
W20260928 05:18:12.538950 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:18:12.531702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48677 (local address 127.0.97.62:35487)
0928 05:18:12.531857 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:12.531861 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:12.532201 (+ 340us) server_negotiation.cc:407] Connection header received
0928 05:18:12.532316 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:12.532319 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:12.532376 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:12.532448 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:18:12.533141 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:12.533810 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:12.534995 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:12.535391 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:12.535473 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:12.538776 (+ 3303us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:48677: BlockingRecv error: recv got EOF from 127.0.97.1:48677 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 28 05:18:12 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:18:12 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:18:12.935504 9628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:12.928023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35487 (local address 127.0.97.2:51047)
0928 05:18:12.928499 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0928 05:18:12.928515 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 05:18:12.928629 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:18:12.928907 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:18:12.928910 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:18:12.929298 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:18:12.929454 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:12.929463 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:12.930366 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:12.930370 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:12.931298 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:12.931309 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:12.932018 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:12.932024 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:12.932174 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:12.932744 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:18:12.932761 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:18:12.935278 (+ 2517us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":274,"threads_started":1}
W20260928 05:18:12.935662 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:18:12.928206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51047 (local address 127.0.97.62:35487)
0928 05:18:12.928417 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:12.928423 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:12.928584 (+ 161us) server_negotiation.cc:407] Connection header received
0928 05:18:12.928718 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:12.928722 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:12.928780 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:12.928877 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:18:12.929560 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:12.930249 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:12.931440 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:12.931869 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:12.931971 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:12.935439 (+ 3468us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:51047: BlockingRecv error: recv got EOF from 127.0.97.2:51047 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 28 05:18:13 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:18:13 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:18:13.348594 9629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:13.341037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35487 (local address 127.0.97.3:57179)
0928 05:18:13.341496 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0928 05:18:13.341514 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 05:18:13.341618 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:18:13.341869 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:18:13.341872 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:18:13.342263 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:18:13.342413 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:13.342421 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:13.343320 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:13.343324 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:13.344218 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:13.344229 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:13.344824 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:13.344827 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:13.344974 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:13.345473 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:18:13.345489 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:18:13.348317 (+ 2828us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":223,"threads_started":1}
W20260928 05:18:13.348781 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:18:13.341131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57179 (local address 127.0.97.62:35487)
0928 05:18:13.341286 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:13.341290 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:13.341578 (+ 288us) server_negotiation.cc:407] Connection header received
0928 05:18:13.341701 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:13.341705 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:13.341765 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:13.341840 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:18:13.342516 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:13.343217 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:13.344319 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:13.344723 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:13.344807 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:13.348527 (+ 3720us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:57179: BlockingRecv error: recv got EOF from 127.0.97.3:57179 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 28 05:18:13 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:18:13 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:18:13.547124 9630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:13.539632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35487 (local address 127.0.97.1:45779)
0928 05:18:13.540086 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0928 05:18:13.540101 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 05:18:13.540208 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:18:13.540457 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:18:13.540460 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:18:13.540803 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:18:13.540999 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:13.541007 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:13.541958 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:13.541961 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:13.542878 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:13.542892 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:13.543696 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:13.543701 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:13.543850 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:13.544444 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:18:13.544460 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:18:13.546916 (+ 2456us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":244,"threads_started":1}
W20260928 05:18:13.547336 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:18:13.539706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45779 (local address 127.0.97.62:35487)
0928 05:18:13.539915 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:13.539921 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:13.540171 (+ 250us) server_negotiation.cc:407] Connection header received
0928 05:18:13.540286 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:13.540289 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:13.540351 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:13.540423 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:18:13.541148 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:13.541862 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:13.543020 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:13.543546 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:13.543669 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:13.547105 (+ 3436us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:45779: BlockingRecv error: recv got EOF from 127.0.97.1:45779 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 28 05:18:13 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:18:13 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:18:13.943715 9631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:13.936337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35487 (local address 127.0.97.2:47535)
0928 05:18:13.936802 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0928 05:18:13.936816 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 05:18:13.936969 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:18:13.937273 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:18:13.937278 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:18:13.937665 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:18:13.937808 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:13.937817 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:13.938756 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:13.938761 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:13.939649 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:13.939659 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:13.940297 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:13.940303 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:13.940459 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:13.941016 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:18:13.941033 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:18:13.943390 (+ 2357us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":253,"threads_started":1}
W20260928 05:18:13.943854 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:18:13.936425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47535 (local address 127.0.97.62:35487)
0928 05:18:13.936608 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:13.936614 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:13.936904 (+ 290us) server_negotiation.cc:407] Connection header received
0928 05:18:13.937088 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:13.937093 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:13.937154 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:13.937232 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:18:13.937937 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:13.938633 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:13.939778 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:13.940163 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:13.940263 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:13.943630 (+ 3367us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:47535: BlockingRecv error: recv got EOF from 127.0.97.2:47535 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 28 05:18:14 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:18:14 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:18:14.357579 9632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:14.349348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35487 (local address 127.0.97.3:55379)
0928 05:18:14.349841 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0928 05:18:14.349859 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 05:18:14.349969 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:18:14.350222 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:18:14.350225 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:18:14.350596 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:18:14.350734 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:14.350742 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:14.351801 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:14.351807 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:14.352754 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:14.352767 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:14.353536 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:14.353541 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:14.353669 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:14.354318 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:18:14.354341 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:18:14.357267 (+ 2926us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":273,"threads_started":1}
W20260928 05:18:14.357669 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:18:14.349432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55379 (local address 127.0.97.62:35487)
0928 05:18:14.349597 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:14.349601 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:14.349931 (+ 330us) server_negotiation.cc:407] Connection header received
0928 05:18:14.350055 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:14.350059 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:14.350121 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:14.350197 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:18:14.350863 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:14.351638 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:14.352911 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:14.353339 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:14.353426 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:14.357448 (+ 4022us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:55379: BlockingRecv error: recv got EOF from 127.0.97.3:55379 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 28 05:18:14 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:18:14 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:18:14.555553 9633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:14.547973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35487 (local address 127.0.97.1:48515)
0928 05:18:14.548514 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0928 05:18:14.548534 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 05:18:14.548647 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:18:14.548953 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:18:14.548956 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:18:14.549366 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:18:14.549528 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:14.549536 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:14.550451 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:14.550455 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:14.551348 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:14.551358 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:14.552045 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:14.552051 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:14.552190 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:14.552710 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:18:14.552727 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:18:14.555236 (+ 2509us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":221,"threads_started":1}
W20260928 05:18:14.555640 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:18:14.548027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48515 (local address 127.0.97.62:35487)
0928 05:18:14.548215 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:14.548222 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:14.548606 (+ 384us) server_negotiation.cc:407] Connection header received
0928 05:18:14.548744 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:14.548748 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:14.548807 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:14.548924 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:18:14.549651 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:14.550344 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:14.551460 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:14.551878 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:14.551983 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:14.555424 (+ 3441us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:48515: BlockingRecv error: recv got EOF from 127.0.97.1:48515 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 28 05:18:14 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:18:14 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:18:14.955106 9634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:14.944545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35487 (local address 127.0.97.2:54827)
0928 05:18:14.945115 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0928 05:18:14.945131 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 05:18:14.945266 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:18:14.945604 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:18:14.945609 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:18:14.946001 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:18:14.946170 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:14.946179 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:14.947211 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:14.947215 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:14.948099 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:14.948116 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:14.948788 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:14.948793 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:14.948977 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:14.952230 (+ 3253us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:18:14.952248 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:18:14.954866 (+ 2618us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":250,"threads_started":1}
W20260928 05:18:14.955264 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:18:14.944604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54827 (local address 127.0.97.62:35487)
0928 05:18:14.944798 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:14.944804 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:14.945215 (+ 411us) server_negotiation.cc:407] Connection header received
0928 05:18:14.945384 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:14.945389 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:14.945477 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:14.945555 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:18:14.946310 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:14.947071 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:14.948239 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:14.948650 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:14.948751 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:14.955045 (+ 6294us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:54827: BlockingRecv error: recv got EOF from 127.0.97.2:54827 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 28 05:18:15 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:18:15 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:18:15.366160 9635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:15.358420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35487 (local address 127.0.97.3:35767)
0928 05:18:15.358996 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0928 05:18:15.359012 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 05:18:15.359142 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:18:15.359429 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:18:15.359434 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:18:15.359826 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:18:15.359991 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:15.360000 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:15.360973 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:15.360977 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:15.361872 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:15.361883 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:15.362532 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:15.362536 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:15.362644 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:15.363143 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:18:15.363160 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:18:15.365918 (+ 2758us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":265,"threads_started":1}
W20260928 05:18:15.366266 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:18:15.358586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35767 (local address 127.0.97.62:35487)
0928 05:18:15.358782 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:15.358789 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:15.359109 (+ 320us) server_negotiation.cc:407] Connection header received
0928 05:18:15.359252 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:15.359256 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:15.359314 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:15.359390 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:18:15.360131 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:15.360821 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:15.361992 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:15.362413 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:15.362506 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:15.366099 (+ 3593us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:35767: BlockingRecv error: recv got EOF from 127.0.97.3:35767 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 28 05:18:15 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:18:15 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:18:15.564071 9636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:15.556453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35487 (local address 127.0.97.1:60787)
0928 05:18:15.556956 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0928 05:18:15.556973 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 05:18:15.557091 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:18:15.557353 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:18:15.557356 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:18:15.557749 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:18:15.557926 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:15.557936 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:15.558952 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:15.558956 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:15.559961 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:15.559973 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:15.560659 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:15.560664 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:15.560788 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:15.561382 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:18:15.561409 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:18:15.563840 (+ 2431us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":262,"threads_started":1}
W20260928 05:18:15.564221 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:18:15.556582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60787 (local address 127.0.97.62:35487)
0928 05:18:15.556741 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:15.556746 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:15.557038 (+ 292us) server_negotiation.cc:407] Connection header received
0928 05:18:15.557178 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:15.557182 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:15.557242 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:15.557318 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:18:15.558046 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:15.558754 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:15.560078 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:15.560511 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:15.560600 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:15.564000 (+ 3400us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:60787: BlockingRecv error: recv got EOF from 127.0.97.1:60787 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 28 05:18:15 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:18:15 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:18:15.963572 9637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:15.955884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35487 (local address 127.0.97.2:45171)
0928 05:18:15.956331 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0928 05:18:15.956347 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 05:18:15.956459 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:18:15.956718 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:18:15.956721 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:18:15.957129 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:18:15.957277 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:15.957285 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:15.958339 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:15.958343 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:15.959241 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:15.959252 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:15.959876 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:15.959880 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:15.960011 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:15.960572 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:18:15.960589 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:18:15.963274 (+ 2685us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":236,"threads_started":1}
W20260928 05:18:15.963619 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:18:15.955956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45171 (local address 127.0.97.62:35487)
0928 05:18:15.956111 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:15.956115 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:15.956411 (+ 296us) server_negotiation.cc:407] Connection header received
0928 05:18:15.956546 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:15.956550 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:15.956611 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:15.956690 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:18:15.957381 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:15.958227 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:15.959346 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:15.959750 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:15.959836 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:15.963440 (+ 3604us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:45171: BlockingRecv error: recv got EOF from 127.0.97.2:45171 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 28 05:18:16 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:18:16 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:18:16.380838 9638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:16.367313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35487 (local address 127.0.97.3:48923)
0928 05:18:16.368445 (+ 1132us) negotiation.cc:107] Waiting for socket to connect
0928 05:18:16.368473 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0928 05:18:16.368682 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:18:16.369462 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:18:16.369468 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:18:16.370175 (+ 707us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:18:16.370702 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:16.370723 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:16.372240 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:16.372244 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:16.373604 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:16.373636 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:16.374864 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:16.374874 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:16.375228 (+ 354us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:16.376462 (+ 1234us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:18:16.376490 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:18:16.380499 (+ 4009us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":904,"thread_start_us":244,"threads_started":1}
W20260928 05:18:16.381011 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:18:16.367752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48923 (local address 127.0.97.62:35487)
0928 05:18:16.367988 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:16.367993 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:16.368686 (+ 693us) server_negotiation.cc:407] Connection header received
0928 05:18:16.369138 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:16.369143 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:16.369223 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:16.369315 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:18:16.371025 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:16.372009 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:16.373914 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:16.374406 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:16.374574 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:16.380743 (+ 6169us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:48923: BlockingRecv error: recv got EOF from 127.0.97.3:48923 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 28 05:18:16 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:18:16 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:18:16.572409 9639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:16.564941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35487 (local address 127.0.97.1:57981)
0928 05:18:16.565400 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0928 05:18:16.565416 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 05:18:16.565525 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:18:16.565779 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:18:16.565782 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:18:16.566152 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:18:16.566331 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:16.566339 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:16.567375 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:16.567380 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:16.568293 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:16.568303 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:16.568976 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:16.568982 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:16.569136 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:16.569630 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:18:16.569646 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:18:16.572177 (+ 2531us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":226,"threads_started":1}
W20260928 05:18:16.572595 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:18:16.564969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57981 (local address 127.0.97.62:35487)
0928 05:18:16.565140 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:16.565144 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:16.565479 (+ 335us) server_negotiation.cc:407] Connection header received
0928 05:18:16.565609 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:16.565613 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:16.565675 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:16.565752 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:18:16.566501 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:16.567249 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:16.568424 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:16.568816 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:16.568952 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:16.572372 (+ 3420us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:57981: BlockingRecv error: recv got EOF from 127.0.97.1:57981 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 28 05:18:16 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:18:16 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:18:16.972025 9640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:16.964442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35487 (local address 127.0.97.2:37095)
0928 05:18:16.964894 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0928 05:18:16.964909 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 05:18:16.965049 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:18:16.965309 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:18:16.965312 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:18:16.965693 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:18:16.965868 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:16.965876 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:16.966779 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:16.966782 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:16.967705 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:16.967717 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:16.968287 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:16.968291 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:16.968403 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:16.968968 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:18:16.968985 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:18:16.971757 (+ 2772us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":245,"threads_started":1}
W20260928 05:18:16.972175 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:18:16.964496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37095 (local address 127.0.97.62:35487)
0928 05:18:16.964713 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:16.964719 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:16.964996 (+ 277us) server_negotiation.cc:407] Connection header received
0928 05:18:16.965142 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:16.965146 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:16.965208 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:16.965282 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:18:16.965982 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:16.966689 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:16.967802 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:16.968189 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:16.968267 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:16.971951 (+ 3684us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:37095: BlockingRecv error: recv got EOF from 127.0.97.2:37095 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 28 05:18:17 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:18:17 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:18:17.389640 9641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:17.381796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35487 (local address 127.0.97.3:52949)
0928 05:18:17.382243 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0928 05:18:17.382259 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 05:18:17.382381 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:18:17.382649 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:18:17.382652 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:18:17.383002 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:18:17.383168 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:17.383176 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:17.384105 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:17.384108 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:17.385307 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:17.385323 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:17.386059 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:17.386065 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:17.386224 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:17.386830 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:18:17.386847 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:18:17.389315 (+ 2468us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":230,"threads_started":1}
W20260928 05:18:17.389722 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:18:17.381868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52949 (local address 127.0.97.62:35487)
0928 05:18:17.382078 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:17.382085 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:17.382420 (+ 335us) server_negotiation.cc:407] Connection header received
0928 05:18:17.382479 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:17.382483 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:17.382544 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:17.382622 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:18:17.383287 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:17.384002 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:17.385432 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:17.385903 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:17.386029 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:17.389523 (+ 3494us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:52949: BlockingRecv error: recv got EOF from 127.0.97.3:52949 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 28 05:18:17 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:18:17 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:18:17.580813 9642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:17.573266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35487 (local address 127.0.97.1:60609)
0928 05:18:17.573728 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0928 05:18:17.573744 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 05:18:17.573847 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:18:17.574117 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:18:17.574121 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:18:17.574494 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:18:17.574630 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:17.574639 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:17.575631 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:17.575635 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:17.576662 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:17.576677 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:17.577322 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:17.577327 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:17.577454 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:17.577946 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:18:17.577962 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:18:17.580513 (+ 2551us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":264,"threads_started":1}
W20260928 05:18:17.580945 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:18:17.573323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60609 (local address 127.0.97.62:35487)
0928 05:18:17.573468 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:17.573472 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:17.573805 (+ 333us) server_negotiation.cc:407] Connection header received
0928 05:18:17.573945 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:17.573949 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:17.574009 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:17.574089 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:18:17.574775 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:17.575515 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:17.576784 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:17.577213 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:17.577298 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:17.580681 (+ 3383us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:60609: BlockingRecv error: recv got EOF from 127.0.97.1:60609 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 28 05:18:17 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:18:17 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:18:17.980531 9643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:17.972964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35487 (local address 127.0.97.2:60319)
0928 05:18:17.973425 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0928 05:18:17.973441 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 05:18:17.973569 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:18:17.973908 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:18:17.973913 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:18:17.974278 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:18:17.974412 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:17.974421 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:17.975398 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:17.975402 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:17.976297 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:17.976308 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:17.976999 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:17.977004 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:17.977165 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:17.977726 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:18:17.977743 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:18:17.980236 (+ 2493us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":267,"threads_started":1}
W20260928 05:18:17.980669 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:18:17.973044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60319 (local address 127.0.97.62:35487)
0928 05:18:17.973220 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:17.973225 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:17.973524 (+ 299us) server_negotiation.cc:407] Connection header received
0928 05:18:17.973693 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:17.973698 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:17.973773 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:17.973859 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:18:17.974556 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:17.975272 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:17.976428 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:17.976838 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:17.976975 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:17.980430 (+ 3455us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:60319: BlockingRecv error: recv got EOF from 127.0.97.2:60319 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 28 05:18:18 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:18:18 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:18:18.398146 9644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:18.390696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35487 (local address 127.0.97.3:47169)
0928 05:18:18.391162 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0928 05:18:18.391177 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 05:18:18.391306 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:18:18.391676 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:18:18.391680 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:18:18.392087 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:18:18.392225 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:18.392233 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:18.393309 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:18.393313 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:18.394232 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:18.394243 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:18.394857 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:18.394861 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:18.394980 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:18.395480 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:18:18.395496 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:18:18.397944 (+ 2448us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":239,"threads_started":1}
W20260928 05:18:18.398325 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:18:18.390821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47169 (local address 127.0.97.62:35487)
0928 05:18:18.391039 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:18.391045 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:18.391252 (+ 207us) server_negotiation.cc:407] Connection header received
0928 05:18:18.391449 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:18.391454 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:18.391537 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:18.391636 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:18:18.392350 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:18.393109 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:18.394339 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:18.394733 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:18.394821 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:18.398105 (+ 3284us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:47169: BlockingRecv error: recv got EOF from 127.0.97.3:47169 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 28 05:18:18 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:18:18 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:18:18.589780 9645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:18.581698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35487 (local address 127.0.97.1:57633)
0928 05:18:18.582172 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0928 05:18:18.582188 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 05:18:18.582294 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:18:18.582560 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:18:18.582563 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:18:18.582951 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:18:18.583116 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:18.583124 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:18.584065 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:18.584073 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:18.585105 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:18.585121 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:18.585920 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:18.585928 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:18.586118 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:18.586678 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:18:18.586695 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:18:18.589551 (+ 2856us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":266,"threads_started":1}
W20260928 05:18:18.589928 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:18:18.581769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57633 (local address 127.0.97.62:35487)
0928 05:18:18.581904 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:18.581908 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:18.582247 (+ 339us) server_negotiation.cc:407] Connection header received
0928 05:18:18.582383 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:18.582387 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:18.582456 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:18.582530 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:18:18.583230 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:18.583908 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:18.585259 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:18.585726 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:18.585866 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:18.589729 (+ 3863us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:57633: BlockingRecv error: recv got EOF from 127.0.97.1:57633 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 28 05:18:18 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:18:18 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:18:18.989496 9646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:18.981462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35487 (local address 127.0.97.2:47515)
0928 05:18:18.981932 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0928 05:18:18.981948 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 05:18:18.982059 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:18:18.982327 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:18:18.982330 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:18:18.982682 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:18:18.982847 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:18.982854 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:18.983838 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:18.983843 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:18.984848 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:18.984893 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:18.985855 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:18.985863 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:18.986058 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:18.986659 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:18:18.986679 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:18:18.989236 (+ 2557us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":253,"threads_started":1}
W20260928 05:18:18.989612 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:18:18.981566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47515 (local address 127.0.97.62:35487)
0928 05:18:18.981708 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:18.981712 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:18.982011 (+ 299us) server_negotiation.cc:407] Connection header received
0928 05:18:18.982152 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:18.982156 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:18.982217 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:18.982298 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:18:18.982951 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:18.983687 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:18.985008 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:18.985652 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:18.985790 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:18.989404 (+ 3614us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:47515: BlockingRecv error: recv got EOF from 127.0.97.2:47515 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 28 05:18:19 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:18:19 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:18:19.407225 9647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:19.398945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35487 (local address 127.0.97.3:59497)
0928 05:18:19.399450 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0928 05:18:19.399467 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 05:18:19.399586 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:18:19.399849 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:18:19.399852 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:18:19.400342 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:18:19.400551 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:19.400561 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:19.401893 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:19.401899 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:19.402870 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:19.402882 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:19.403563 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:19.403569 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:19.403708 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:19.404229 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:18:19.404248 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:18:19.406903 (+ 2655us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":281,"threads_started":1}
W20260928 05:18:19.407325 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:18:19.399060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59497 (local address 127.0.97.62:35487)
0928 05:18:19.399248 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:19.399254 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:19.399539 (+ 285us) server_negotiation.cc:407] Connection header received
0928 05:18:19.399676 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:19.399680 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:19.399741 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:19.399817 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:18:19.400712 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:19.401752 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:19.403003 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:19.403427 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:19.403530 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:19.407113 (+ 3583us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:59497: BlockingRecv error: recv got EOF from 127.0.97.3:59497 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 28 05:18:19 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:18:19 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:18:19.603261 9648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:19.590594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35487 (local address 127.0.97.1:58433)
0928 05:18:19.591023 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0928 05:18:19.591040 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 05:18:19.591166 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:18:19.591482 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:18:19.591487 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:18:19.591856 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:18:19.592006 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:19.592015 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:19.592972 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:19.592978 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:19.593858 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:19.593869 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:19.594546 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:19.594553 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:19.594715 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:19.600362 (+ 5647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:18:19.600380 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:18:19.602949 (+ 2569us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":263,"threads_started":1}
W20260928 05:18:19.603361 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:18:19.590682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58433 (local address 127.0.97.62:35487)
0928 05:18:19.590852 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:19.590858 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:19.591114 (+ 256us) server_negotiation.cc:407] Connection header received
0928 05:18:19.591278 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:19.591283 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:19.591356 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:19.591431 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:18:19.592133 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:19.592825 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:19.593988 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:19.594420 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:19.594530 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:19.603150 (+ 8620us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:58433: BlockingRecv error: recv got EOF from 127.0.97.1:58433 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 28 05:18:19 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:18:20 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:18:20.002265 9649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:19.990396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35487 (local address 127.0.97.2:32813)
0928 05:18:19.990859 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0928 05:18:19.990873 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 05:18:19.991023 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:18:19.991314 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:18:19.991318 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:18:19.991653 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:18:19.991801 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:19.991809 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:19.992758 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:19.992763 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:19.993676 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:19.993687 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:19.994313 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:19.994318 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:19.994476 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:19.999252 (+ 4776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:18:19.999271 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:18:20.001958 (+ 2687us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":264,"threads_started":1}
W20260928 05:18:20.002355 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:18:19.990481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:32813 (local address 127.0.97.62:35487)
0928 05:18:19.990673 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:19.990679 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:19.990975 (+ 296us) server_negotiation.cc:407] Connection header received
0928 05:18:19.991137 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:19.991141 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:19.991198 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:19.991279 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:18:19.991932 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:19.992629 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:19.993807 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:19.994190 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:19.994290 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:20.002155 (+ 7865us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:32813: BlockingRecv error: recv got EOF from 127.0.97.2:32813 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 28 05:18:20 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:18:20 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:18:20.415846 9650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:20.408130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35487 (local address 127.0.97.3:47267)
0928 05:18:20.408574 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0928 05:18:20.408589 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 05:18:20.408720 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:18:20.409170 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:18:20.409175 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:18:20.409631 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:18:20.409777 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:20.409786 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:20.410754 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:20.410758 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:20.411754 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:20.411766 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:20.412397 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:20.412403 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:20.412566 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:20.413109 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:18:20.413125 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:18:20.415546 (+ 2421us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":245,"threads_started":1}
W20260928 05:18:20.415979 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:18:20.408171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47267 (local address 127.0.97.62:35487)
0928 05:18:20.408339 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:20.408345 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:20.408667 (+ 322us) server_negotiation.cc:407] Connection header received
0928 05:18:20.408903 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:20.408910 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:20.409030 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:20.409127 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:18:20.409906 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:20.410624 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:20.411878 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:20.412274 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:20.412373 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:20.415753 (+ 3380us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:47267: BlockingRecv error: recv got EOF from 127.0.97.3:47267 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 28 05:18:20 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:18:20 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:18:20.612305 9651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:20.604182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35487 (local address 127.0.97.1:41373)
0928 05:18:20.604646 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0928 05:18:20.604662 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 05:18:20.604769 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:18:20.605145 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:18:20.605149 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:18:20.605508 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:18:20.605747 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:20.605759 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:20.607048 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:20.607054 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:20.607963 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:20.607974 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:20.608695 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:20.608700 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:20.608824 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:20.609379 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:18:20.609396 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:18:20.612086 (+ 2690us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":221,"threads_started":1}
W20260928 05:18:20.612578 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:18:20.604249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41373 (local address 127.0.97.62:35487)
0928 05:18:20.604416 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:20.604421 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:20.604725 (+ 304us) server_negotiation.cc:407] Connection header received
0928 05:18:20.604892 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:20.604898 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:20.604998 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:20.605108 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:18:20.605882 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:20.606929 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:20.608097 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:20.608518 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:20.608614 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:20.612312 (+ 3698us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:41373: BlockingRecv error: recv got EOF from 127.0.97.1:41373 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 28 05:18:21 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:18:21 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:18:21.011116 9652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:21.003124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35487 (local address 127.0.97.2:58669)
0928 05:18:21.003607 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0928 05:18:21.003627 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 05:18:21.003749 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:18:21.004074 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:18:21.004078 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:18:21.004479 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:18:21.004644 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:21.004653 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:21.005663 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:21.005667 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:21.006618 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:21.006629 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:21.007266 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:21.007269 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:21.007376 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:21.008030 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:18:21.008047 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:18:21.010857 (+ 2810us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":229,"threads_started":1}
W20260928 05:18:21.011210 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:18:21.003297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58669 (local address 127.0.97.62:35487)
0928 05:18:21.003489 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:21.003495 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:21.003697 (+ 202us) server_negotiation.cc:407] Connection header received
0928 05:18:21.003841 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:21.003844 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:21.003903 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:21.003998 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:18:21.004778 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:21.005537 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:21.006732 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:21.007162 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:21.007268 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:21.011041 (+ 3773us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:58669: BlockingRecv error: recv got EOF from 127.0.97.2:58669 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Sep 28 05:18:21 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:18:21 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:18:21.424265 9653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:21.416748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35487 (local address 127.0.97.3:36357)
0928 05:18:21.417272 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0928 05:18:21.417286 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 05:18:21.417409 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:18:21.417698 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:18:21.417703 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:18:21.418090 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:18:21.418237 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:21.418245 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:21.419193 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:21.419197 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:21.420083 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:21.420094 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:21.420766 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:21.420771 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:21.420925 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:21.421440 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:18:21.421457 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:18:21.423966 (+ 2509us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":276,"threads_started":1}
W20260928 05:18:21.424371 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:18:21.416761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36357 (local address 127.0.97.62:35487)
0928 05:18:21.416962 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:21.416968 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:21.417368 (+ 400us) server_negotiation.cc:407] Connection header received
0928 05:18:21.417516 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:21.417521 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:21.417587 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:21.417661 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:18:21.418364 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:21.419071 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:21.420211 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:21.420609 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:21.420708 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:21.424169 (+ 3461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:36357: BlockingRecv error: recv got EOF from 127.0.97.3:36357 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 28 05:18:21 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:18:21 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:18:21.620471 9654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:21.613212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35487 (local address 127.0.97.1:50587)
0928 05:18:21.613592 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0928 05:18:21.613607 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 05:18:21.613715 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:18:21.613962 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:18:21.613965 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:18:21.614279 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:18:21.614443 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:21.614452 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:21.615333 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:21.615338 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:21.616255 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:21.616267 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:21.616919 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:21.616924 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:21.617039 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:21.617549 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:18:21.617565 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:18:21.620149 (+ 2584us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":193,"threads_started":1}
W20260928 05:18:21.620560 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:18:21.613260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50587 (local address 127.0.97.62:35487)
0928 05:18:21.613552 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:21.613558 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:21.613669 (+ 111us) server_negotiation.cc:407] Connection header received
0928 05:18:21.613796 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:21.613799 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:21.613868 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:21.613947 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:18:21.614541 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:21.615227 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:21.616372 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:21.616782 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:21.616896 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:21.620337 (+ 3441us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:50587: BlockingRecv error: recv got EOF from 127.0.97.1:50587 (error 108)
Metrics: {"server-negotiator.queue_time_us":153}
Sep 28 05:18:22 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:18:22 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:18:22.019804 9655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:22.011910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35487 (local address 127.0.97.2:34237)
0928 05:18:22.012367 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0928 05:18:22.012382 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 05:18:22.012495 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:18:22.012753 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:18:22.012756 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:18:22.013135 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:18:22.013303 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:22.013315 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:22.014405 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:22.014410 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:22.015546 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:22.015558 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:22.016234 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:22.016237 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:22.016347 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:22.016918 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:18:22.016935 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:18:22.019514 (+ 2579us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":223,"threads_started":1}
W20260928 05:18:22.019893 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:18:22.012003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34237 (local address 127.0.97.62:35487)
0928 05:18:22.012148 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:22.012153 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:22.012445 (+ 292us) server_negotiation.cc:407] Connection header received
0928 05:18:22.012586 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:22.012589 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:22.012648 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:22.012724 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:18:22.013429 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:22.014294 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:22.015688 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:22.016130 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:22.016223 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:22.019686 (+ 3463us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:34237: BlockingRecv error: recv got EOF from 127.0.97.2:34237 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 28 05:18:22 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:18:22 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:18:22.432466 9656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:22.425134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35487 (local address 127.0.97.3:52855)
0928 05:18:22.425576 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0928 05:18:22.425590 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 05:18:22.425713 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:18:22.426018 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:18:22.426023 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:18:22.426360 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:18:22.426523 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:22.426533 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:22.427495 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:22.427500 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:22.428424 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:22.428436 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:22.429117 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:22.429122 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:22.429262 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:22.429806 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:18:22.429824 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:18:22.432150 (+ 2326us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":253,"threads_started":1}
W20260928 05:18:22.432592 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:18:22.425205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52855 (local address 127.0.97.62:35487)
0928 05:18:22.425422 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:22.425428 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:22.425665 (+ 237us) server_negotiation.cc:407] Connection header received
0928 05:18:22.425831 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:22.425834 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:22.425892 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:22.425969 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:18:22.426653 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:22.427360 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:22.428551 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:22.428981 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:22.429080 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:22.432378 (+ 3298us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:52855: BlockingRecv error: recv got EOF from 127.0.97.3:52855 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 28 05:18:22 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:18:22 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:18:22.629144 9657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:22.621424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35487 (local address 127.0.97.1:59577)
0928 05:18:22.621867 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0928 05:18:22.621882 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 05:18:22.622002 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:18:22.622321 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:18:22.622324 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:18:22.622686 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:18:22.622859 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:22.622867 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:22.623783 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:22.623787 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:22.624702 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:22.624714 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:22.625593 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:22.625599 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:22.625792 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:22.626374 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:18:22.626397 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:18:22.628887 (+ 2490us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":244,"threads_started":1}
W20260928 05:18:22.629235 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:18:22.621498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59577 (local address 127.0.97.62:35487)
0928 05:18:22.621667 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:22.621671 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:22.621966 (+ 295us) server_negotiation.cc:407] Connection header received
0928 05:18:22.622129 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:22.622133 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:22.622211 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:22.622294 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:18:22.622976 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:22.623667 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:22.624838 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:22.625425 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:22.625573 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:22.629051 (+ 3478us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:59577: BlockingRecv error: recv got EOF from 127.0.97.1:59577 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 28 05:18:23 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:18:23 dist-test-slave-whsn krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:18:23.029305 9658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:23.020947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35487 (local address 127.0.97.2:55193)
0928 05:18:23.021413 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0928 05:18:23.021452 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 05:18:23.021574 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:18:23.021891 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:18:23.021896 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:18:23.022371 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:18:23.022575 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:23.022587 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:23.023577 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:23.023582 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:23.025042 (+ 1460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:23.025053 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:23.025686 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:23.025690 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:23.025792 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:23.026330 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:18:23.026346 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:18:23.028989 (+ 2643us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35487: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":218,"threads_started":1}
W20260928 05:18:23.029402 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:18:23.021020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55193 (local address 127.0.97.62:35487)
0928 05:18:23.021212 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:23.021218 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:23.021538 (+ 320us) server_negotiation.cc:407] Connection header received
0928 05:18:23.021696 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:23.021701 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:23.021767 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:23.021844 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:18:23.022702 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:23.023443 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:23.025151 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:23.025571 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:23.025657 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:23.029169 (+ 3512us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:55193: BlockingRecv error: recv got EOF from 127.0.97.2:55193 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 05:18:23.282327 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 9200
I20260928 05:18:23.293074 9660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:23.283214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34992 (local address 127.0.97.1:32827)
0928 05:18:23.283687 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:23.283693 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:23.283718 (+ 25us) server_negotiation.cc:407] Connection header received
0928 05:18:23.283773 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:23.283779 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:23.283978 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:23.284127 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:18:23.285428 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:23.286549 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:23.287765 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:23.288277 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:23.288367 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:23.290530 (+ 2163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:18:23.290538 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:18:23.290550 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:18:23.290577 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:18:23.292602 (+ 2025us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0928 05:18:23.292812 (+ 210us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34992: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":234,"threads_started":1}
W20260928 05:18:23.293169 9660 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34992: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260928 05:18:23.293226 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 9200: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:32827: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260928 05:18:23.293406 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 9200
I20260928 05:18:23.306279 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 9351
I20260928 05:18:23.316664 9661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:23.307024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57394 (local address 127.0.97.2:36179)
0928 05:18:23.307584 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:23.307591 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:23.307620 (+ 29us) server_negotiation.cc:407] Connection header received
0928 05:18:23.307669 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:23.307675 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:23.307892 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:23.308038 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:18:23.309062 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:23.310065 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:23.311230 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:23.311768 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:23.311864 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:23.313958 (+ 2094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:18:23.313967 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:18:23.313979 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:18:23.314012 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:18:23.316226 (+ 2214us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0928 05:18:23.316440 (+ 214us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57394: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":277,"threads_started":1}
W20260928 05:18:23.316737 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 9351: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:36179: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260928 05:18:23.316913 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 9351
W20260928 05:18:23.316838 9661 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57394: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260928 05:18:23.329686 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 9482
I20260928 05:18:23.340979 9662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:23.330396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39088 (local address 127.0.97.3:39055)
0928 05:18:23.330862 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:23.330868 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:23.330890 (+ 22us) server_negotiation.cc:407] Connection header received
0928 05:18:23.330941 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:23.330947 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:23.331156 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:23.331307 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:18:23.332685 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:23.334095 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:23.335362 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:23.335877 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:23.335983 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:23.338344 (+ 2361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:18:23.338352 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:18:23.338364 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:18:23.338401 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:18:23.340379 (+ 1978us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0928 05:18:23.340665 (+ 286us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39088: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":223,"threads_started":1}
W20260928 05:18:23.341017 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 9482: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:39055: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260928 05:18:23.341061 9662 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39088: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260928 05:18:23.341132 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 9482
I20260928 05:18:23.353856 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 9131
I20260928 05:18:23.354908 9193 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:18:23.463389 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 9131
2026-09-28T05:18:23Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (20452 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-28T05:18:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:18:23Z Disabled control of system clock
I20260928 05:18:23.526165 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572622576160-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572622576160-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572622576160-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42037
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:45271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42037
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 05:18:23.819423 9667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:18:23.819669 9667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:18:23.819761 9667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:18:23.826568 9667 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:18:23.826651 9667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:18:23.826704 9667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:18:23.826777 9667 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:18:23.826829 9667 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:18:23.835297 9667 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572622576160-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572622576160-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42037
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42037
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572622576160-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:18:23.836387 9667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:18:23.838037 9667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:18:23.848712 9673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:18:23.848798 9672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:18:23.848757 9675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:18:23.849305 9667 server_base.cc:1034] running on GCE node
I20260928 05:18:23.849951 9667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:18:23.851150 9667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:18:23.852322 9667 hybrid_clock.cc:648] HybridClock initialized: now 1790572703852287 us; error 29 us; skew 500 ppm
I20260928 05:18:23.852629 9667 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 05:18:23.854631 9667 webserver.cc:460] Webserver started at http://127.0.97.62:44673/ using document root <none> and password file <none>
I20260928 05:18:23.855232 9667 fs_manager.cc:359] Metadata directory not provided
I20260928 05:18:23.855317 9667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:18:23.855618 9667 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:18:23.858256 9667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572622576160-388-0/minicluster-data/master-0/data/instance:
uuid: "ba3994d27e314ed28203af8bc017655f"
format_stamp: "Formatted at 2026-09-28 05:18:23 on dist-test-slave-whsn"
server_key: "efc5526147ba0361bf3018c45e1edc60"
server_key_iv: "72fe586f69822bcbd090d3939f8adbfa"
server_key_version: "encryptionkey@0"
I20260928 05:18:23.858958 9667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572622576160-388-0/minicluster-data/master-0/wal/instance:
uuid: "ba3994d27e314ed28203af8bc017655f"
format_stamp: "Formatted at 2026-09-28 05:18:23 on dist-test-slave-whsn"
server_key: "efc5526147ba0361bf3018c45e1edc60"
server_key_iv: "72fe586f69822bcbd090d3939f8adbfa"
server_key_version: "encryptionkey@0"
I20260928 05:18:23.863440 9667 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 05:18:23.866412 9681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:18:23.867563 9667 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 05:18:23.867699 9667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572622576160-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572622576160-388-0/minicluster-data/master-0/wal
uuid: "ba3994d27e314ed28203af8bc017655f"
format_stamp: "Formatted at 2026-09-28 05:18:23 on dist-test-slave-whsn"
server_key: "efc5526147ba0361bf3018c45e1edc60"
server_key_iv: "72fe586f69822bcbd090d3939f8adbfa"
server_key_version: "encryptionkey@0"
I20260928 05:18:23.867872 9667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572622576160-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572622576160-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572622576160-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:18:23.884617 9667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:18:23.885352 9667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:18:23.885591 9667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:18:23.900892 9667 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42037
I20260928 05:18:23.900911 9732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42037 every 8 connection(s)
I20260928 05:18:23.902055 9667 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:18:23.905282 9733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:18:23.910686 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 9667
I20260928 05:18:23.910892 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572622576160-388-0/minicluster-data/master-0/wal/instance
I20260928 05:18:23.911625 388 external_mini_cluster.cc:1428] Setting key c5ef784b6d90294b951a32ee7434f64a
I20260928 05:18:23.915481 9733 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba3994d27e314ed28203af8bc017655f: Bootstrap starting.
I20260928 05:18:23.918193 9733 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba3994d27e314ed28203af8bc017655f: Neither blocks nor log segments found. Creating new log.
I20260928 05:18:23.919067 9733 log.cc:824] T 00000000000000000000000000000000 P ba3994d27e314ed28203af8bc017655f: Log is configured to *not* fsync() on all Append() calls
I20260928 05:18:23.921402 9733 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba3994d27e314ed28203af8bc017655f: No bootstrap required, opened a new log
I20260928 05:18:23.923558 9736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:23.913533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37750 (local address 127.0.97.62:42037)
0928 05:18:23.914387 (+ 854us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:23.914397 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:23.914437 (+ 40us) server_negotiation.cc:407] Connection header received
0928 05:18:23.915988 (+ 1551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:23.916008 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:23.916387 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:23.916723 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:18:23.917257 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:23.918480 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:23.919700 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:23.920339 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:23.920487 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:23.921499 (+ 1012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:18:23.921508 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:18:23.921526 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:18:23.922032 (+ 506us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:18:23.922099 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:18:23.922117 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:18:23.922208 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 05:18:23.922423 (+ 215us) server_negotiation.cc:299] Negotiation successful
0928 05:18:23.922683 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":305,"threads_started":1}
I20260928 05:18:23.928329 9733 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba3994d27e314ed28203af8bc017655f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba3994d27e314ed28203af8bc017655f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42037 } }
I20260928 05:18:23.928624 9733 raft_consensus.cc:374] T 00000000000000000000000000000000 P ba3994d27e314ed28203af8bc017655f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:18:23.928728 9733 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba3994d27e314ed28203af8bc017655f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba3994d27e314ed28203af8bc017655f, State: Initialized, Role: FOLLOWER
I20260928 05:18:23.929236 9733 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba3994d27e314ed28203af8bc017655f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba3994d27e314ed28203af8bc017655f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42037 } }
I20260928 05:18:23.929426 9733 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba3994d27e314ed28203af8bc017655f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:18:23.929544 9733 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba3994d27e314ed28203af8bc017655f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:18:23.929677 9733 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba3994d27e314ed28203af8bc017655f [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:18:23.931825 9733 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba3994d27e314ed28203af8bc017655f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba3994d27e314ed28203af8bc017655f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42037 } }
I20260928 05:18:23.932222 9733 leader_election.cc:302] T 00000000000000000000000000000000 P ba3994d27e314ed28203af8bc017655f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba3994d27e314ed28203af8bc017655f; no voters:
I20260928 05:18:23.932674 9733 leader_election.cc:288] T 00000000000000000000000000000000 P ba3994d27e314ed28203af8bc017655f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:18:23.932788 9738 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba3994d27e314ed28203af8bc017655f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:18:23.933410 9738 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba3994d27e314ed28203af8bc017655f [term 1 LEADER]: Becoming Leader. State: Replica: ba3994d27e314ed28203af8bc017655f, State: Running, Role: LEADER
I20260928 05:18:23.933878 9738 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba3994d27e314ed28203af8bc017655f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba3994d27e314ed28203af8bc017655f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42037 } }
I20260928 05:18:23.934288 9733 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba3994d27e314ed28203af8bc017655f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:18:23.936329 9740 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba3994d27e314ed28203af8bc017655f [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba3994d27e314ed28203af8bc017655f. Latest consensus state: current_term: 1 leader_uuid: "ba3994d27e314ed28203af8bc017655f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba3994d27e314ed28203af8bc017655f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42037 } } }
I20260928 05:18:23.936762 9740 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba3994d27e314ed28203af8bc017655f [sys.catalog]: This master's current role is: LEADER
I20260928 05:18:23.936659 9739 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba3994d27e314ed28203af8bc017655f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ba3994d27e314ed28203af8bc017655f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba3994d27e314ed28203af8bc017655f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42037 } } }
I20260928 05:18:23.936959 9739 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba3994d27e314ed28203af8bc017655f [sys.catalog]: This master's current role is: LEADER
I20260928 05:18:23.937558 9747 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:18:23.942122 9747 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:18:23.950353 9747 catalog_manager.cc:1347] Generated new cluster ID: 199157dc5dfb4d1d912dab903ba7b78b
I20260928 05:18:23.950433 9747 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:18:23.970979 9747 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:18:23.972222 9747 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:18:23.983959 9747 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ba3994d27e314ed28203af8bc017655f: Generated new TSK 0
I20260928 05:18:23.984730 9747 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 05:18:23.993587 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572622576160-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572622576160-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572622576160-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572622576160-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42037
--builtin_ntp_servers=127.0.97.20:45271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260928 05:18:24.329726 9757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:18:24.329975 9757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:18:24.330060 9757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:18:24.338245 9757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:18:24.338361 9757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:18:24.338486 9757 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 05:18:24.347568 9757 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572622576160-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572622576160-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572622576160-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42037
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572622576160-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:18:24.348650 9757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:18:24.350289 9757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:18:24.361773 9762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:18:24.361845 9763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:18:24.362035 9765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:18:24.362664 9757 server_base.cc:1034] running on GCE node
I20260928 05:18:24.363231 9757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:18:24.364105 9757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:18:24.365285 9757 hybrid_clock.cc:648] HybridClock initialized: now 1790572704365227 us; error 50 us; skew 500 ppm
I20260928 05:18:24.365569 9757 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 05:18:24.367657 9757 webserver.cc:460] Webserver started at http://127.0.97.1:43769/ using document root <none> and password file <none>
I20260928 05:18:24.368247 9757 fs_manager.cc:359] Metadata directory not provided
I20260928 05:18:24.368331 9757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:18:24.368626 9757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:18:24.371275 9757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572622576160-388-0/minicluster-data/ts-0/data/instance:
uuid: "c45f5dc7bf3842c1b8fe5c4e31bc5435"
format_stamp: "Formatted at 2026-09-28 05:18:24 on dist-test-slave-whsn"
server_key: "63dcbb112f8ec67ad6aace9a338db2b8"
server_key_iv: "934628c756d53456feecb9c56c53cead"
server_key_version: "encryptionkey@0"
I20260928 05:18:24.372013 9757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572622576160-388-0/minicluster-data/ts-0/wal/instance:
uuid: "c45f5dc7bf3842c1b8fe5c4e31bc5435"
format_stamp: "Formatted at 2026-09-28 05:18:24 on dist-test-slave-whsn"
server_key: "63dcbb112f8ec67ad6aace9a338db2b8"
server_key_iv: "934628c756d53456feecb9c56c53cead"
server_key_version: "encryptionkey@0"
I20260928 05:18:24.376605 9757 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.007s
I20260928 05:18:24.379736 9771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:18:24.380956 9757 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 05:18:24.381089 9757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572622576160-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572622576160-388-0/minicluster-data/ts-0/wal
uuid: "c45f5dc7bf3842c1b8fe5c4e31bc5435"
format_stamp: "Formatted at 2026-09-28 05:18:24 on dist-test-slave-whsn"
server_key: "63dcbb112f8ec67ad6aace9a338db2b8"
server_key_iv: "934628c756d53456feecb9c56c53cead"
server_key_version: "encryptionkey@0"
I20260928 05:18:24.381264 9757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572622576160-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572622576160-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572622576160-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:18:24.396216 9757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:18:24.396991 9757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:18:24.397219 9757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:18:24.398099 9757 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:18:24.399425 9757 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:18:24.399514 9757 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:18:24.399616 9757 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:18:24.399681 9757 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:18:24.427361 9757 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44459
I20260928 05:18:24.427392 9883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44459 every 8 connection(s)
I20260928 05:18:24.428498 9757 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572622576160-388-0/minicluster-data/ts-0/data/info.pb
I20260928 05:18:24.435839 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 9757
I20260928 05:18:24.436017 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572622576160-388-0/minicluster-data/ts-0/wal/instance
I20260928 05:18:24.436574 388 external_mini_cluster.cc:1428] Setting key 49f6913b05a4ec50fc80e4b019a79892
I20260928 05:18:24.439015 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572622576160-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572622576160-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572622576160-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572622576160-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42037
--builtin_ntp_servers=127.0.97.20:45271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 05:18:24.439926 9886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:24.430727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58299 (local address 127.0.97.62:42037)
0928 05:18:24.431102 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:24.431107 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:24.432206 (+ 1099us) server_negotiation.cc:407] Connection header received
0928 05:18:24.433853 (+ 1647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:24.433858 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:24.433930 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:24.434017 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:18:24.434978 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:24.435829 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:24.437523 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:24.437937 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:24.438040 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:24.439206 (+ 1166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:18:24.439210 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:18:24.439213 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:18:24.439229 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:18:24.439262 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:18:24.439272 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:18:24.439341 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 05:18:24.439531 (+ 190us) server_negotiation.cc:299] Negotiation successful
0928 05:18:24.439613 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":193,"threads_started":1}
I20260928 05:18:24.440768 9887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:24.431268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42037 (local address 127.0.97.1:58299)
0928 05:18:24.432072 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0928 05:18:24.432110 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 05:18:24.433623 (+ 1513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:18:24.434233 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:18:24.434241 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:18:24.434321 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:18:24.434821 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:24.434836 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:24.435964 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:24.435968 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:24.437403 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:24.437416 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:24.438129 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:24.438136 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:24.438359 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:24.438943 (+ 584us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 05:18:24.438945 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:18:24.438979 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 05:18:24.439044 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:18:24.439351 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:18:24.439360 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:18:24.439368 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 05:18:24.439511 (+ 143us) client_negotiation.cc:240] Negotiation successful
0928 05:18:24.439839 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":245,"threads_started":1}
I20260928 05:18:24.442112 9884 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42037
I20260928 05:18:24.442348 9884 heartbeater.cc:464] Registering TS with master...
I20260928 05:18:24.442996 9884 heartbeater.cc:511] Master 127.0.97.62:42037 requested a full tablet report, sending...
I20260928 05:18:24.444962 9698 ts_manager.cc:194] Registered new tserver with Master: c45f5dc7bf3842c1b8fe5c4e31bc5435 (127.0.97.1:44459)
I20260928 05:18:24.446436 9698 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:58299
W20260928 05:18:24.736435 9888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:18:24.736678 9888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:18:24.736795 9888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:18:24.743532 9888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:18:24.743620 9888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:18:24.743759 9888 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 05:18:24.751991 9888 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572622576160-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572622576160-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572622576160-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42037
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572622576160-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:18:24.753062 9888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:18:24.754621 9888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:18:24.765791 9896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:18:24.765849 9893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:18:24.765842 9894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:18:24.766773 9888 server_base.cc:1034] running on GCE node
I20260928 05:18:24.767257 9888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:18:24.768059 9888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:18:24.769227 9888 hybrid_clock.cc:648] HybridClock initialized: now 1790572704769192 us; error 27 us; skew 500 ppm
I20260928 05:18:24.769490 9888 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 05:18:24.771392 9888 webserver.cc:460] Webserver started at http://127.0.97.2:46239/ using document root <none> and password file <none>
I20260928 05:18:24.771973 9888 fs_manager.cc:359] Metadata directory not provided
I20260928 05:18:24.772058 9888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:18:24.772339 9888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:18:24.774854 9888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572622576160-388-0/minicluster-data/ts-1/data/instance:
uuid: "2464c2ce43254d8fb4b5e8963774d211"
format_stamp: "Formatted at 2026-09-28 05:18:24 on dist-test-slave-whsn"
server_key: "750dbab1e3b378118a4f2ec97a7234bd"
server_key_iv: "1f41f43f39acf3b8d529c407a67e1cf0"
server_key_version: "encryptionkey@0"
I20260928 05:18:24.775532 9888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572622576160-388-0/minicluster-data/ts-1/wal/instance:
uuid: "2464c2ce43254d8fb4b5e8963774d211"
format_stamp: "Formatted at 2026-09-28 05:18:24 on dist-test-slave-whsn"
server_key: "750dbab1e3b378118a4f2ec97a7234bd"
server_key_iv: "1f41f43f39acf3b8d529c407a67e1cf0"
server_key_version: "encryptionkey@0"
I20260928 05:18:24.779948 9888 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260928 05:18:24.782898 9902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:18:24.784036 9888 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 05:18:24.784161 9888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572622576160-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572622576160-388-0/minicluster-data/ts-1/wal
uuid: "2464c2ce43254d8fb4b5e8963774d211"
format_stamp: "Formatted at 2026-09-28 05:18:24 on dist-test-slave-whsn"
server_key: "750dbab1e3b378118a4f2ec97a7234bd"
server_key_iv: "1f41f43f39acf3b8d529c407a67e1cf0"
server_key_version: "encryptionkey@0"
I20260928 05:18:24.784327 9888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572622576160-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572622576160-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572622576160-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:18:24.795068 9888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:18:24.795724 9888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:18:24.795948 9888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:18:24.796756 9888 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:18:24.798084 9888 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:18:24.798161 9888 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:18:24.798250 9888 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:18:24.798321 9888 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:18:24.826186 9888 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42135
I20260928 05:18:24.826241 10014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42135 every 8 connection(s)
I20260928 05:18:24.827334 9888 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572622576160-388-0/minicluster-data/ts-1/data/info.pb
I20260928 05:18:24.837523 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 9888
I20260928 05:18:24.837726 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572622576160-388-0/minicluster-data/ts-1/wal/instance
I20260928 05:18:24.838286 388 external_mini_cluster.cc:1428] Setting key 5f27909bc999523ba06504e350581e97
I20260928 05:18:24.838380 9886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:24.829560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37937 (local address 127.0.97.62:42037)
0928 05:18:24.829729 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:24.829733 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:24.830974 (+ 1241us) server_negotiation.cc:407] Connection header received
0928 05:18:24.832543 (+ 1569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:24.832548 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:24.832620 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:24.832702 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:18:24.833684 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:24.834429 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:24.835936 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:24.836301 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:24.836401 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:24.837697 (+ 1296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:18:24.837702 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:18:24.837705 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:18:24.837721 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:18:24.837750 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:18:24.837762 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:18:24.837854 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 05:18:24.838044 (+ 190us) server_negotiation.cc:299] Negotiation successful
0928 05:18:24.838110 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 05:18:24.839397 10017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:24.830016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42037 (local address 127.0.97.2:37937)
0928 05:18:24.830820 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0928 05:18:24.830858 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 05:18:24.832303 (+ 1445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:18:24.832945 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:18:24.832952 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:18:24.833037 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:18:24.833537 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:24.833551 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:24.834572 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:24.834576 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:24.835807 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:24.835817 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:24.836493 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:24.836501 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:24.836727 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:24.837340 (+ 613us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 05:18:24.837344 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:18:24.837396 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 05:18:24.837498 (+ 102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:18:24.837857 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:18:24.837867 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:18:24.837874 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 05:18:24.838046 (+ 172us) client_negotiation.cc:240] Negotiation successful
0928 05:18:24.838370 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":248,"threads_started":1}
I20260928 05:18:24.840605 10015 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42037
I20260928 05:18:24.840807 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572622576160-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572622576160-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572622576160-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572622576160-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42037
--builtin_ntp_servers=127.0.97.20:45271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 05:18:24.840835 10015 heartbeater.cc:464] Registering TS with master...
I20260928 05:18:24.841514 10015 heartbeater.cc:511] Master 127.0.97.62:42037 requested a full tablet report, sending...
I20260928 05:18:24.842660 9698 ts_manager.cc:194] Registered new tserver with Master: 2464c2ce43254d8fb4b5e8963774d211 (127.0.97.2:42135)
I20260928 05:18:24.843376 9698 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:37937
W20260928 05:18:25.151533 10018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:18:25.151789 10018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:18:25.151916 10018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:18:25.158910 10018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:18:25.158998 10018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:18:25.159137 10018 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 05:18:25.167690 10018 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572622576160-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572622576160-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572622576160-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42037
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572622576160-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:18:25.168898 10018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:18:25.170590 10018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:18:25.182904 10023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:18:25.182978 10024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:18:25.182978 10026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:18:25.183826 10018 server_base.cc:1034] running on GCE node
I20260928 05:18:25.184355 10018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:18:25.185263 10018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:18:25.186455 10018 hybrid_clock.cc:648] HybridClock initialized: now 1790572705186410 us; error 43 us; skew 500 ppm
I20260928 05:18:25.186756 10018 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 05:18:25.192965 10018 webserver.cc:460] Webserver started at http://127.0.97.3:43311/ using document root <none> and password file <none>
I20260928 05:18:25.193624 10018 fs_manager.cc:359] Metadata directory not provided
I20260928 05:18:25.193722 10018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:18:25.194038 10018 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:18:25.196751 10018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572622576160-388-0/minicluster-data/ts-2/data/instance:
uuid: "1bf47670864e422a8141beef3294a0e6"
format_stamp: "Formatted at 2026-09-28 05:18:25 on dist-test-slave-whsn"
server_key: "f55362eb24b15b0db9bf29f384083153"
server_key_iv: "49f766603d11e9de53e5be4fd3f357d1"
server_key_version: "encryptionkey@0"
I20260928 05:18:25.197552 10018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572622576160-388-0/minicluster-data/ts-2/wal/instance:
uuid: "1bf47670864e422a8141beef3294a0e6"
format_stamp: "Formatted at 2026-09-28 05:18:25 on dist-test-slave-whsn"
server_key: "f55362eb24b15b0db9bf29f384083153"
server_key_iv: "49f766603d11e9de53e5be4fd3f357d1"
server_key_version: "encryptionkey@0"
I20260928 05:18:25.202311 10018 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 05:18:25.205606 10032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:18:25.206800 10018 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 05:18:25.206946 10018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572622576160-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572622576160-388-0/minicluster-data/ts-2/wal
uuid: "1bf47670864e422a8141beef3294a0e6"
format_stamp: "Formatted at 2026-09-28 05:18:25 on dist-test-slave-whsn"
server_key: "f55362eb24b15b0db9bf29f384083153"
server_key_iv: "49f766603d11e9de53e5be4fd3f357d1"
server_key_version: "encryptionkey@0"
I20260928 05:18:25.207118 10018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572622576160-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572622576160-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572622576160-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:18:25.224553 10018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:18:25.225282 10018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:18:25.225518 10018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:18:25.226333 10018 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:18:25.227594 10018 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:18:25.227686 10018 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:18:25.227780 10018 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:18:25.227845 10018 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:18:25.256567 10018 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33375
I20260928 05:18:25.256613 10144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33375 every 8 connection(s)
I20260928 05:18:25.257787 10018 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572622576160-388-0/minicluster-data/ts-2/data/info.pb
I20260928 05:18:25.259335 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 10018
I20260928 05:18:25.259480 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572622576160-388-0/minicluster-data/ts-2/wal/instance
I20260928 05:18:25.259864 388 external_mini_cluster.cc:1428] Setting key df7948c10e9b7127939503d9ae221b79
I20260928 05:18:25.269114 9886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:25.260044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51639 (local address 127.0.97.62:42037)
0928 05:18:25.260261 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:25.260267 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:25.261767 (+ 1500us) server_negotiation.cc:407] Connection header received
0928 05:18:25.263494 (+ 1727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:25.263500 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:25.263578 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:25.263665 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:18:25.264596 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:25.265337 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:25.266823 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:25.267215 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:25.267308 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:25.268454 (+ 1146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:18:25.268458 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:18:25.268462 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:18:25.268476 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:18:25.268502 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:18:25.268512 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:18:25.268571 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 05:18:25.268732 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 05:18:25.268795 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 05:18:25.269999 10147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:25.260598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42037 (local address 127.0.97.3:51639)
0928 05:18:25.261621 (+ 1023us) negotiation.cc:107] Waiting for socket to connect
0928 05:18:25.261665 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 05:18:25.263212 (+ 1547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:18:25.263845 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:18:25.263852 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:18:25.263936 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:18:25.264469 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:25.264485 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:25.265474 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:25.265478 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:25.266697 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:25.266713 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:25.267377 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:25.267390 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:25.267595 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:25.268190 (+ 595us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 05:18:25.268192 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:18:25.268226 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 05:18:25.268290 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:18:25.268580 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:18:25.268585 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:18:25.268590 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 05:18:25.268722 (+ 132us) client_negotiation.cc:240] Negotiation successful
0928 05:18:25.269028 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":330,"threads_started":1}
I20260928 05:18:25.271168 10145 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42037
I20260928 05:18:25.271431 10145 heartbeater.cc:464] Registering TS with master...
I20260928 05:18:25.272140 10145 heartbeater.cc:511] Master 127.0.97.62:42037 requested a full tablet report, sending...
I20260928 05:18:25.273418 9697 ts_manager.cc:194] Registered new tserver with Master: 1bf47670864e422a8141beef3294a0e6 (127.0.97.3:33375)
I20260928 05:18:25.274325 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 05:18:25.274420 9697 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:51639
W20260928 05:18:25.280364 9886 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:18:25.279021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37754 (local address 127.0.97.62:42037)
0928 05:18:25.279159 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:25.279162 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:25.279623 (+ 461us) server_negotiation.cc:407] Connection header received
0928 05:18:25.279727 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:25.279731 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:25.279792 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:25.279871 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:18:25.280146 (+ 275us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37754: BlockingRecv error: recv got EOF from 127.0.0.1:37754 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 05:18:25.281126 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 9757
I20260928 05:18:25.288563 10154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:25.282015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60746 (local address 127.0.97.1:44459)
0928 05:18:25.282708 (+ 693us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:25.282714 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:25.282736 (+ 22us) server_negotiation.cc:407] Connection header received
0928 05:18:25.282791 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:25.282797 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:25.282950 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:25.283057 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:18:25.283371 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:25.284335 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:25.285656 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:25.286153 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:25.286248 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:25.287265 (+ 1017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:18:25.287272 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:18:25.287282 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:18:25.287844 (+ 562us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:18:25.287892 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:18:25.287913 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:18:25.287995 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 05:18:25.288130 (+ 135us) server_negotiation.cc:299] Negotiation successful
0928 05:18:25.288237 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":473,"threads_started":1}
I20260928 05:18:25.289677 9879 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:18:25.369035 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 9757
I20260928 05:18:25.381855 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 9888
I20260928 05:18:25.389272 10156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:25.382597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59414 (local address 127.0.97.2:42135)
0928 05:18:25.383076 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:25.383082 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:25.383107 (+ 25us) server_negotiation.cc:407] Connection header received
0928 05:18:25.383159 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:25.383165 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:25.383366 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:25.383506 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:18:25.383872 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:25.384845 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:25.386004 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:25.386522 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:25.386628 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:25.387646 (+ 1018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:18:25.387655 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:18:25.387668 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:18:25.388344 (+ 676us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:18:25.388398 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:18:25.388420 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:18:25.388520 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 05:18:25.388642 (+ 122us) server_negotiation.cc:299] Negotiation successful
0928 05:18:25.388776 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":228,"threads_started":1}
I20260928 05:18:25.390527 10010 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:18:25.468367 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 9888
I20260928 05:18:25.481514 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 10018
I20260928 05:18:25.489288 10158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:25.482490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41596 (local address 127.0.97.3:33375)
0928 05:18:25.483116 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:25.483123 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:25.483150 (+ 27us) server_negotiation.cc:407] Connection header received
0928 05:18:25.483204 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:25.483210 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:25.483417 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:25.483557 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:18:25.483975 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:25.485067 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:25.486244 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:25.486771 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:25.486880 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:25.487856 (+ 976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:18:25.487864 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:18:25.487877 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:18:25.488465 (+ 588us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:18:25.488511 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:18:25.488529 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:18:25.488625 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 05:18:25.488767 (+ 142us) server_negotiation.cc:299] Negotiation successful
0928 05:18:25.488927 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":331,"threads_started":1}
I20260928 05:18:25.490477 10140 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:18:25.568573 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 10018
I20260928 05:18:25.580937 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 9667
I20260928 05:18:25.581966 9728 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:18:25.678468 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 9667
2026-09-28T05:18:25Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2211 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-28T05:18:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:18:25Z Disabled control of system clock
I20260928 05:18:25.737193 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572622576160-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572622576160-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572622576160-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38619
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:35491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38619
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 05:18:26.029627 10164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:18:26.029875 10164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:18:26.030040 10164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:18:26.036727 10164 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:18:26.036813 10164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:18:26.036888 10164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:18:26.036973 10164 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:18:26.037027 10164 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:18:26.045481 10164 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572622576160-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572622576160-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38619
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38619
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572622576160-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:18:26.046572 10164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:18:26.048167 10164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:18:26.058543 10170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:18:26.058737 10169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:18:26.058712 10172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:18:26.059511 10164 server_base.cc:1034] running on GCE node
I20260928 05:18:26.060199 10164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:18:26.061483 10164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:18:26.062666 10164 hybrid_clock.cc:648] HybridClock initialized: now 1790572706062621 us; error 35 us; skew 500 ppm
I20260928 05:18:26.062963 10164 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 05:18:26.065066 10164 webserver.cc:460] Webserver started at http://127.0.97.62:45271/ using document root <none> and password file <none>
I20260928 05:18:26.065680 10164 fs_manager.cc:359] Metadata directory not provided
I20260928 05:18:26.065773 10164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:18:26.066061 10164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:18:26.068676 10164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572622576160-388-0/minicluster-data/master-0/data/instance:
uuid: "8613216e2ec2405893e0d54f515efddd"
format_stamp: "Formatted at 2026-09-28 05:18:26 on dist-test-slave-whsn"
server_key: "58ac984c60c3243e520e9b7eef78d73d"
server_key_iv: "ca100709b1bef9a654490f55a21d790b"
server_key_version: "encryptionkey@0"
I20260928 05:18:26.069417 10164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572622576160-388-0/minicluster-data/master-0/wal/instance:
uuid: "8613216e2ec2405893e0d54f515efddd"
format_stamp: "Formatted at 2026-09-28 05:18:26 on dist-test-slave-whsn"
server_key: "58ac984c60c3243e520e9b7eef78d73d"
server_key_iv: "ca100709b1bef9a654490f55a21d790b"
server_key_version: "encryptionkey@0"
I20260928 05:18:26.073911 10164 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 05:18:26.077052 10178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:18:26.078342 10164 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:18:26.078509 10164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572622576160-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572622576160-388-0/minicluster-data/master-0/wal
uuid: "8613216e2ec2405893e0d54f515efddd"
format_stamp: "Formatted at 2026-09-28 05:18:26 on dist-test-slave-whsn"
server_key: "58ac984c60c3243e520e9b7eef78d73d"
server_key_iv: "ca100709b1bef9a654490f55a21d790b"
server_key_version: "encryptionkey@0"
I20260928 05:18:26.078704 10164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572622576160-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572622576160-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572622576160-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:18:26.084177 10164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:18:26.085006 10164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:18:26.085281 10164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:18:26.102074 10164 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38619
I20260928 05:18:26.102105 10229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38619 every 8 connection(s)
I20260928 05:18:26.103308 10164 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:18:26.106722 10230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:18:26.111464 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 10164
I20260928 05:18:26.111629 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572622576160-388-0/minicluster-data/master-0/wal/instance
I20260928 05:18:26.112228 388 external_mini_cluster.cc:1428] Setting key 7286b2664ae90e147824b154c552fd17
I20260928 05:18:26.116521 10230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8613216e2ec2405893e0d54f515efddd: Bootstrap starting.
I20260928 05:18:26.119632 10233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:26.114027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48534 (local address 127.0.97.62:38619)
0928 05:18:26.114982 (+ 955us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:26.114992 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:26.115031 (+ 39us) server_negotiation.cc:407] Connection header received
0928 05:18:26.116538 (+ 1507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:26.116564 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:26.116999 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:26.117284 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:18:26.118243 (+ 959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:18:26.118248 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:18:26.118260 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:18:26.118273 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:18:26.118327 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:18:26.118341 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:18:26.118502 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0928 05:18:26.118591 (+ 89us) server_negotiation.cc:299] Negotiation successful
0928 05:18:26.118814 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":328,"threads_started":1}
I20260928 05:18:26.119670 10230 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8613216e2ec2405893e0d54f515efddd: Neither blocks nor log segments found. Creating new log.
I20260928 05:18:26.120867 10230 log.cc:824] T 00000000000000000000000000000000 P 8613216e2ec2405893e0d54f515efddd: Log is configured to *not* fsync() on all Append() calls
I20260928 05:18:26.123739 10230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8613216e2ec2405893e0d54f515efddd: No bootstrap required, opened a new log
I20260928 05:18:26.130853 10230 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8613216e2ec2405893e0d54f515efddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8613216e2ec2405893e0d54f515efddd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38619 } }
I20260928 05:18:26.131179 10230 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8613216e2ec2405893e0d54f515efddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:18:26.131270 10230 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8613216e2ec2405893e0d54f515efddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8613216e2ec2405893e0d54f515efddd, State: Initialized, Role: FOLLOWER
I20260928 05:18:26.131784 10230 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8613216e2ec2405893e0d54f515efddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8613216e2ec2405893e0d54f515efddd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38619 } }
I20260928 05:18:26.132015 10230 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8613216e2ec2405893e0d54f515efddd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:18:26.132140 10230 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8613216e2ec2405893e0d54f515efddd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:18:26.132296 10230 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8613216e2ec2405893e0d54f515efddd [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:18:26.134678 10230 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8613216e2ec2405893e0d54f515efddd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8613216e2ec2405893e0d54f515efddd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38619 } }
I20260928 05:18:26.135131 10230 leader_election.cc:302] T 00000000000000000000000000000000 P 8613216e2ec2405893e0d54f515efddd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8613216e2ec2405893e0d54f515efddd; no voters:
I20260928 05:18:26.135637 10230 leader_election.cc:288] T 00000000000000000000000000000000 P 8613216e2ec2405893e0d54f515efddd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:18:26.135720 10235 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8613216e2ec2405893e0d54f515efddd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:18:26.136368 10235 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8613216e2ec2405893e0d54f515efddd [term 1 LEADER]: Becoming Leader. State: Replica: 8613216e2ec2405893e0d54f515efddd, State: Running, Role: LEADER
I20260928 05:18:26.136812 10235 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8613216e2ec2405893e0d54f515efddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8613216e2ec2405893e0d54f515efddd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38619 } }
I20260928 05:18:26.137575 10230 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8613216e2ec2405893e0d54f515efddd [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:18:26.139273 10237 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8613216e2ec2405893e0d54f515efddd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8613216e2ec2405893e0d54f515efddd. Latest consensus state: current_term: 1 leader_uuid: "8613216e2ec2405893e0d54f515efddd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8613216e2ec2405893e0d54f515efddd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38619 } } }
I20260928 05:18:26.139690 10237 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8613216e2ec2405893e0d54f515efddd [sys.catalog]: This master's current role is: LEADER
I20260928 05:18:26.140761 10236 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8613216e2ec2405893e0d54f515efddd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8613216e2ec2405893e0d54f515efddd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8613216e2ec2405893e0d54f515efddd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38619 } } }
I20260928 05:18:26.141094 10236 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8613216e2ec2405893e0d54f515efddd [sys.catalog]: This master's current role is: LEADER
I20260928 05:18:26.142334 10244 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:18:26.146862 10244 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:18:26.155092 10244 catalog_manager.cc:1347] Generated new cluster ID: 211c384621e34c958087aea77d19052c
I20260928 05:18:26.155193 10244 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:18:26.181751 10244 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:18:26.182210 10244 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:18:26.189633 10244 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8613216e2ec2405893e0d54f515efddd: Generated new TSK 0
I20260928 05:18:26.190531 10244 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 05:18:26.199071 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572622576160-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572622576160-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572622576160-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572622576160-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38619
--builtin_ntp_servers=127.0.97.20:35491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 05:18:26.505314 10254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:18:26.505563 10254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:18:26.505684 10254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:18:26.512641 10254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:18:26.512727 10254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:18:26.512871 10254 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 05:18:26.521567 10254 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572622576160-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572622576160-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572622576160-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38619
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572622576160-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:18:26.522647 10254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:18:26.524276 10254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:18:26.536002 10259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:18:26.536043 10262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:18:26.536064 10260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:18:26.536756 10254 server_base.cc:1034] running on GCE node
I20260928 05:18:26.537333 10254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:18:26.538268 10254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:18:26.539470 10254 hybrid_clock.cc:648] HybridClock initialized: now 1790572706539435 us; error 31 us; skew 500 ppm
I20260928 05:18:26.539738 10254 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 05:18:26.541812 10254 webserver.cc:460] Webserver started at http://127.0.97.1:38135/ using document root <none> and password file <none>
I20260928 05:18:26.542440 10254 fs_manager.cc:359] Metadata directory not provided
I20260928 05:18:26.542526 10254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:18:26.542796 10254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:18:26.545475 10254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572622576160-388-0/minicluster-data/ts-0/data/instance:
uuid: "ad7b69ff90284d52aaa9a959abeb8952"
format_stamp: "Formatted at 2026-09-28 05:18:26 on dist-test-slave-whsn"
server_key: "3c0c882fec5d5bbd3caad267c0854929"
server_key_iv: "e8f6016b79763ab826d5a3ca30839438"
server_key_version: "encryptionkey@0"
I20260928 05:18:26.546171 10254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572622576160-388-0/minicluster-data/ts-0/wal/instance:
uuid: "ad7b69ff90284d52aaa9a959abeb8952"
format_stamp: "Formatted at 2026-09-28 05:18:26 on dist-test-slave-whsn"
server_key: "3c0c882fec5d5bbd3caad267c0854929"
server_key_iv: "e8f6016b79763ab826d5a3ca30839438"
server_key_version: "encryptionkey@0"
I20260928 05:18:26.550733 10254 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260928 05:18:26.553717 10268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:18:26.554869 10254 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 05:18:26.554991 10254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572622576160-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572622576160-388-0/minicluster-data/ts-0/wal
uuid: "ad7b69ff90284d52aaa9a959abeb8952"
format_stamp: "Formatted at 2026-09-28 05:18:26 on dist-test-slave-whsn"
server_key: "3c0c882fec5d5bbd3caad267c0854929"
server_key_iv: "e8f6016b79763ab826d5a3ca30839438"
server_key_version: "encryptionkey@0"
I20260928 05:18:26.555184 10254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572622576160-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572622576160-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572622576160-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:18:26.560070 10254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:18:26.560750 10254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:18:26.560985 10254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:18:26.561810 10254 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:18:26.563002 10254 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:18:26.563083 10254 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:18:26.563175 10254 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:18:26.563244 10254 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:18:26.591192 10254 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38373
I20260928 05:18:26.591261 10380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38373 every 8 connection(s)
I20260928 05:18:26.592344 10254 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572622576160-388-0/minicluster-data/ts-0/data/info.pb
I20260928 05:18:26.596534 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 10254
I20260928 05:18:26.596721 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572622576160-388-0/minicluster-data/ts-0/wal/instance
I20260928 05:18:26.597630 388 external_mini_cluster.cc:1428] Setting key 1626a205c67771971680f84deaaf6303
I20260928 05:18:26.600520 10233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:26.594678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36111 (local address 127.0.97.62:38619)
0928 05:18:26.594859 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:26.594864 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:26.596172 (+ 1308us) server_negotiation.cc:407] Connection header received
0928 05:18:26.598461 (+ 2289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:26.598467 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:26.598520 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:26.598598 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:18:26.599918 (+ 1320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:18:26.599922 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:18:26.599926 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:18:26.599946 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:18:26.599976 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:18:26.599987 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:18:26.600076 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 05:18:26.600298 (+ 222us) server_negotiation.cc:299] Negotiation successful
0928 05:18:26.600338 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 05:18:26.600556 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572622576160-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572622576160-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572622576160-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572622576160-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38619
--builtin_ntp_servers=127.0.97.20:35491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260928 05:18:26.601806 10383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:26.595136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38619 (local address 127.0.97.1:36111)
0928 05:18:26.596015 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0928 05:18:26.596061 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 05:18:26.598187 (+ 2126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:18:26.598829 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:18:26.598837 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:18:26.598929 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:18:26.599601 (+ 672us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 05:18:26.599604 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:18:26.599654 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 05:18:26.599759 (+ 105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:18:26.600078 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:18:26.600088 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:18:26.600095 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 05:18:26.600410 (+ 315us) client_negotiation.cc:240] Negotiation successful
0928 05:18:26.600650 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":273,"threads_started":1}
I20260928 05:18:26.603077 10381 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38619
I20260928 05:18:26.603372 10381 heartbeater.cc:464] Registering TS with master...
I20260928 05:18:26.603999 10381 heartbeater.cc:511] Master 127.0.97.62:38619 requested a full tablet report, sending...
I20260928 05:18:26.605944 10195 ts_manager.cc:194] Registered new tserver with Master: ad7b69ff90284d52aaa9a959abeb8952 (127.0.97.1:38373)
W20260928 05:18:26.913736 10384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:18:26.914007 10384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:18:26.914147 10384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:18:26.921197 10384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:18:26.921291 10384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:18:26.921452 10384 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 05:18:26.930190 10384 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572622576160-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572622576160-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572622576160-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38619
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572622576160-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:18:26.931448 10384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:18:26.933243 10384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:18:26.945437 10390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:18:26.945482 10392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:18:26.945940 10389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:18:26.946343 10384 server_base.cc:1034] running on GCE node
I20260928 05:18:26.946862 10384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:18:26.947731 10384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:18:26.948969 10384 hybrid_clock.cc:648] HybridClock initialized: now 1790572706948917 us; error 39 us; skew 500 ppm
I20260928 05:18:26.949301 10384 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 05:18:26.951331 10384 webserver.cc:460] Webserver started at http://127.0.97.2:43581/ using document root <none> and password file <none>
I20260928 05:18:26.951972 10384 fs_manager.cc:359] Metadata directory not provided
I20260928 05:18:26.952064 10384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:18:26.952353 10384 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:18:26.955118 10384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572622576160-388-0/minicluster-data/ts-1/data/instance:
uuid: "6c4aa13f5a29456bbb5abe287374e7ea"
format_stamp: "Formatted at 2026-09-28 05:18:26 on dist-test-slave-whsn"
server_key: "e987cf0a9d517fffc93c8f36768dd2cf"
server_key_iv: "18d874916723b42deefad8bb319fa00a"
server_key_version: "encryptionkey@0"
I20260928 05:18:26.955875 10384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572622576160-388-0/minicluster-data/ts-1/wal/instance:
uuid: "6c4aa13f5a29456bbb5abe287374e7ea"
format_stamp: "Formatted at 2026-09-28 05:18:26 on dist-test-slave-whsn"
server_key: "e987cf0a9d517fffc93c8f36768dd2cf"
server_key_iv: "18d874916723b42deefad8bb319fa00a"
server_key_version: "encryptionkey@0"
I20260928 05:18:26.960891 10384 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260928 05:18:26.964012 10398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:18:26.965248 10384 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.003s
I20260928 05:18:26.965387 10384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572622576160-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572622576160-388-0/minicluster-data/ts-1/wal
uuid: "6c4aa13f5a29456bbb5abe287374e7ea"
format_stamp: "Formatted at 2026-09-28 05:18:26 on dist-test-slave-whsn"
server_key: "e987cf0a9d517fffc93c8f36768dd2cf"
server_key_iv: "18d874916723b42deefad8bb319fa00a"
server_key_version: "encryptionkey@0"
I20260928 05:18:26.965591 10384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572622576160-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572622576160-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572622576160-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:18:26.970703 10384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:18:26.971387 10384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:18:26.971621 10384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:18:26.972419 10384 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:18:26.973677 10384 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:18:26.973766 10384 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:18:26.973865 10384 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:18:26.973932 10384 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:18:27.002365 10384 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45867
I20260928 05:18:27.002421 10510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45867 every 8 connection(s)
I20260928 05:18:27.003609 10384 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572622576160-388-0/minicluster-data/ts-1/data/info.pb
I20260928 05:18:27.010010 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 10384
I20260928 05:18:27.010171 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572622576160-388-0/minicluster-data/ts-1/wal/instance
I20260928 05:18:27.010612 388 external_mini_cluster.cc:1428] Setting key c3ade520b77b55d5e316a51c5ca7f8e5
I20260928 05:18:27.011243 10233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:27.005988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43475 (local address 127.0.97.62:38619)
0928 05:18:27.006196 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:27.006203 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:27.007510 (+ 1307us) server_negotiation.cc:407] Connection header received
0928 05:18:27.009132 (+ 1622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:27.009136 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:27.009187 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:27.009276 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:18:27.010562 (+ 1286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:18:27.010568 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:18:27.010572 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:18:27.010591 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:18:27.010631 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:18:27.010646 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:18:27.010775 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0928 05:18:27.010939 (+ 164us) server_negotiation.cc:299] Negotiation successful
0928 05:18:27.010998 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 05:18:27.012205 10513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:27.006446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38619 (local address 127.0.97.2:43475)
0928 05:18:27.007362 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0928 05:18:27.007398 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 05:18:27.008826 (+ 1428us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:18:27.009455 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:18:27.009462 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:18:27.009550 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:18:27.010262 (+ 712us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 05:18:27.010266 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:18:27.010311 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 05:18:27.010392 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:18:27.010763 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:18:27.010771 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:18:27.010776 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 05:18:27.010984 (+ 208us) client_negotiation.cc:240] Negotiation successful
0928 05:18:27.011195 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":278,"threads_started":1}
I20260928 05:18:27.013492 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572622576160-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572622576160-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572622576160-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572622576160-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38619
--builtin_ntp_servers=127.0.97.20:35491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260928 05:18:27.013518 10511 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38619
I20260928 05:18:27.013795 10511 heartbeater.cc:464] Registering TS with master...
I20260928 05:18:27.014321 10511 heartbeater.cc:511] Master 127.0.97.62:38619 requested a full tablet report, sending...
I20260928 05:18:27.015595 10195 ts_manager.cc:194] Registered new tserver with Master: 6c4aa13f5a29456bbb5abe287374e7ea (127.0.97.2:45867)
W20260928 05:18:27.308175 10514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:18:27.308405 10514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:18:27.308496 10514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:18:27.315188 10514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:18:27.315268 10514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:18:27.315374 10514 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 05:18:27.323619 10514 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572622576160-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572622576160-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572622576160-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38619
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572622576160-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:18:27.324628 10514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:18:27.326256 10514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:18:27.337574 10519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:18:27.337601 10522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:18:27.337628 10520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:18:27.338765 10514 server_base.cc:1034] running on GCE node
I20260928 05:18:27.339424 10514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:18:27.340356 10514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:18:27.341538 10514 hybrid_clock.cc:648] HybridClock initialized: now 1790572707341488 us; error 45 us; skew 500 ppm
I20260928 05:18:27.341802 10514 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 05:18:27.343678 10514 webserver.cc:460] Webserver started at http://127.0.97.3:36325/ using document root <none> and password file <none>
I20260928 05:18:27.344309 10514 fs_manager.cc:359] Metadata directory not provided
I20260928 05:18:27.344398 10514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:18:27.344691 10514 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:18:27.347908 10514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572622576160-388-0/minicluster-data/ts-2/data/instance:
uuid: "2a91ea8eb2334912878dd7489b765189"
format_stamp: "Formatted at 2026-09-28 05:18:27 on dist-test-slave-whsn"
server_key: "25b51a2e52347c39ce391e73d543ffb1"
server_key_iv: "84e9344d37d083cf0ca201a7aaab0e0a"
server_key_version: "encryptionkey@0"
I20260928 05:18:27.348747 10514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572622576160-388-0/minicluster-data/ts-2/wal/instance:
uuid: "2a91ea8eb2334912878dd7489b765189"
format_stamp: "Formatted at 2026-09-28 05:18:27 on dist-test-slave-whsn"
server_key: "25b51a2e52347c39ce391e73d543ffb1"
server_key_iv: "84e9344d37d083cf0ca201a7aaab0e0a"
server_key_version: "encryptionkey@0"
I20260928 05:18:27.353272 10514 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 05:18:27.356258 10528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:18:27.357435 10514 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 05:18:27.357582 10514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572622576160-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572622576160-388-0/minicluster-data/ts-2/wal
uuid: "2a91ea8eb2334912878dd7489b765189"
format_stamp: "Formatted at 2026-09-28 05:18:27 on dist-test-slave-whsn"
server_key: "25b51a2e52347c39ce391e73d543ffb1"
server_key_iv: "84e9344d37d083cf0ca201a7aaab0e0a"
server_key_version: "encryptionkey@0"
I20260928 05:18:27.357769 10514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572622576160-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572622576160-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572622576160-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:18:27.362850 10514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:18:27.363514 10514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:18:27.363725 10514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:18:27.364507 10514 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:18:27.365768 10514 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:18:27.365855 10514 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:18:27.365952 10514 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:18:27.366020 10514 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:18:27.391858 10514 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34673
I20260928 05:18:27.391928 10640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34673 every 8 connection(s)
I20260928 05:18:27.393070 10514 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572622576160-388-0/minicluster-data/ts-2/data/info.pb
I20260928 05:18:27.400063 10233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:27.395326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59625 (local address 127.0.97.62:38619)
0928 05:18:27.395533 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:27.395539 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:27.396817 (+ 1278us) server_negotiation.cc:407] Connection header received
0928 05:18:27.398351 (+ 1534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:27.398355 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:27.398402 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:27.398473 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:18:27.399578 (+ 1105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:18:27.399582 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:18:27.399585 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:18:27.399595 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:18:27.399617 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:18:27.399627 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:18:27.399702 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 05:18:27.399831 (+ 129us) server_negotiation.cc:299] Negotiation successful
0928 05:18:27.399872 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 05:18:27.400900 10643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:27.395833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38619 (local address 127.0.97.3:59625)
0928 05:18:27.396683 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0928 05:18:27.396720 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 05:18:27.398119 (+ 1399us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:18:27.398648 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:18:27.398656 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:18:27.398738 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:18:27.399335 (+ 597us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 05:18:27.399338 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:18:27.399372 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 05:18:27.399460 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:18:27.399692 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:18:27.399697 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:18:27.399702 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 05:18:27.399887 (+ 185us) client_negotiation.cc:240] Negotiation successful
0928 05:18:27.400062 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":258,"threads_started":1}
I20260928 05:18:27.401140 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 10514
I20260928 05:18:27.401278 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572622576160-388-0/minicluster-data/ts-2/wal/instance
I20260928 05:18:27.401783 388 external_mini_cluster.cc:1428] Setting key 0f9f3004781e5613e4133459ff69d59b
I20260928 05:18:27.402118 10641 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38619
I20260928 05:18:27.402376 10641 heartbeater.cc:464] Registering TS with master...
I20260928 05:18:27.403066 10641 heartbeater.cc:511] Master 127.0.97.62:38619 requested a full tablet report, sending...
I20260928 05:18:27.404402 10194 ts_manager.cc:194] Registered new tserver with Master: 2a91ea8eb2334912878dd7489b765189 (127.0.97.3:34673)
I20260928 05:18:27.416522 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 05:18:27.421988 10233 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:18:27.420473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48542 (local address 127.0.97.62:38619)
0928 05:18:27.420609 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:27.420614 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:27.421120 (+ 506us) server_negotiation.cc:407] Connection header received
0928 05:18:27.421261 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:27.421267 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:27.421316 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:27.421390 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:18:27.421728 (+ 338us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48542: BlockingRecv error: recv got EOF from 127.0.0.1:48542 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 05:18:27.422358 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 10254
I20260928 05:18:27.425602 10650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:27.423177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37138 (local address 127.0.97.1:38373)
0928 05:18:27.423643 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:27.423649 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:27.423670 (+ 21us) server_negotiation.cc:407] Connection header received
0928 05:18:27.423717 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:27.423722 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:27.423895 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:27.424033 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:18:27.424937 (+ 904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:18:27.424945 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:18:27.424956 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:18:27.424980 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:18:27.425017 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:18:27.425031 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:18:27.425141 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 05:18:27.425271 (+ 130us) server_negotiation.cc:299] Negotiation successful
0928 05:18:27.425360 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":225,"threads_started":1}
I20260928 05:18:27.426604 10376 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:18:27.504128 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 10254
I20260928 05:18:27.516842 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 10384
I20260928 05:18:27.520255 10652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:27.517689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52378 (local address 127.0.97.2:45867)
0928 05:18:27.518170 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:27.518177 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:27.518202 (+ 25us) server_negotiation.cc:407] Connection header received
0928 05:18:27.518267 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:27.518273 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:27.518482 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:27.518621 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:18:27.519537 (+ 916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:18:27.519542 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:18:27.519553 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:18:27.519570 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:18:27.519607 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:18:27.519619 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:18:27.519732 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 05:18:27.519861 (+ 129us) server_negotiation.cc:299] Negotiation successful
0928 05:18:27.519968 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":242,"threads_started":1}
I20260928 05:18:27.521473 10506 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:18:27.599057 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 10384
I20260928 05:18:27.611357 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 10514
I20260928 05:18:27.614923 10654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:27.612185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42992 (local address 127.0.97.3:34673)
0928 05:18:27.612837 (+ 652us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:27.612843 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:27.612908 (+ 65us) server_negotiation.cc:407] Connection header received
0928 05:18:27.612969 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:27.612975 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:27.613213 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:27.613332 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:18:27.614280 (+ 948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:18:27.614286 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:18:27.614297 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:18:27.614314 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:18:27.614353 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:18:27.614366 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:18:27.614493 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 05:18:27.614543 (+ 50us) server_negotiation.cc:299] Negotiation successful
0928 05:18:27.614633 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":352,"threads_started":1}
I20260928 05:18:27.616021 10636 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:18:27.693923 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 10514
I20260928 05:18:27.706328 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 10164
I20260928 05:18:27.707345 10225 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:18:27.803148 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 10164
2026-09-28T05:18:27Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2123 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572622576160-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:18:27 dist-test-slave-whsn krb5kdc[10659](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:18:27 dist-test-slave-whsn krb5kdc[10659](info): set up 2 sockets
Sep 28 05:18:27 dist-test-slave-whsn krb5kdc[10659](info): commencing operation
krb5kdc: starting...
W20260928 05:18:28.797462 388 mini_kdc.cc:121] Time spent starting KDC: real 0.960s user 0.004s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572622576160-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572622576160-388-0/krb5kdc/test-user.keytab.
Sep 28 05:18:28 dist-test-slave-whsn krb5kdc[10659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572708, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:18:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:18:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 05:18:28.982033 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572622576160-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572622576160-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572622576160-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46109
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46109
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:18:29.277742 10672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:18:29.277983 10672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:18:29.278111 10672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:18:29.284866 10672 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:18:29.284961 10672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:18:29.285032 10672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:18:29.285111 10672 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:18:29.285171 10672 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:18:29.294178 10672 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572622576160-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572622576160-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46109
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572622576160-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:18:29.295310 10672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:18:29.296468 10672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:18:29.306555 10677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:18:29.306632 10680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:18:29.306653 10678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:18:29.307253 10672 server_base.cc:1034] running on GCE node
I20260928 05:18:29.307871 10672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:18:29.309043 10672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:18:29.310227 10672 hybrid_clock.cc:648] HybridClock initialized: now 1790572709310182 us; error 36 us; skew 500 ppm
I20260928 05:18:29.310508 10672 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:18:29 dist-test-slave-whsn krb5kdc[10659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572709, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:18:29.313191 10672 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 05:18:29.314378 10672 webserver.cc:460] Webserver started at http://127.0.97.62:33151/ using document root <none> and password file <none>
I20260928 05:18:29.314927 10672 fs_manager.cc:359] Metadata directory not provided
I20260928 05:18:29.315016 10672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:18:29.315232 10672 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:18:29.317860 10672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572622576160-388-0/minicluster-data/master-0/data/instance:
uuid: "c47a8cee0bfb4f858d7bf08b5b105c2a"
format_stamp: "Formatted at 2026-09-28 05:18:29 on dist-test-slave-whsn"
server_key: "30fbc02c07aed5ae91ffbf3b3a60be1d"
server_key_iv: "7bb1a950db22065f3f9ac3471c297550"
server_key_version: "encryptionkey@0"
I20260928 05:18:29.318495 10672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572622576160-388-0/minicluster-data/master-0/wal/instance:
uuid: "c47a8cee0bfb4f858d7bf08b5b105c2a"
format_stamp: "Formatted at 2026-09-28 05:18:29 on dist-test-slave-whsn"
server_key: "30fbc02c07aed5ae91ffbf3b3a60be1d"
server_key_iv: "7bb1a950db22065f3f9ac3471c297550"
server_key_version: "encryptionkey@0"
I20260928 05:18:29.322829 10672 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260928 05:18:29.325774 10687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:18:29.326891 10672 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 05:18:29.327035 10672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572622576160-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572622576160-388-0/minicluster-data/master-0/wal
uuid: "c47a8cee0bfb4f858d7bf08b5b105c2a"
format_stamp: "Formatted at 2026-09-28 05:18:29 on dist-test-slave-whsn"
server_key: "30fbc02c07aed5ae91ffbf3b3a60be1d"
server_key_iv: "7bb1a950db22065f3f9ac3471c297550"
server_key_version: "encryptionkey@0"
I20260928 05:18:29.327183 10672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572622576160-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572622576160-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572622576160-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:18:29.351617 10672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:18:29.356357 10672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:18:29.356652 10672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:18:29.374756 10672 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46109
I20260928 05:18:29.374778 10738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46109 every 8 connection(s)
I20260928 05:18:29.375977 10672 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:18:29.376477 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 10672
I20260928 05:18:29.376636 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572622576160-388-0/minicluster-data/master-0/wal/instance
I20260928 05:18:29.377271 388 external_mini_cluster.cc:1428] Setting key 1ad1ea062d84ff84bbd59511104a9437
I20260928 05:18:29.380123 10739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 05:18:29 dist-test-slave-whsn krb5kdc[10659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572708, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:18:29.392272 10739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c47a8cee0bfb4f858d7bf08b5b105c2a: Bootstrap starting.
I20260928 05:18:29.395445 10739 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c47a8cee0bfb4f858d7bf08b5b105c2a: Neither blocks nor log segments found. Creating new log.
I20260928 05:18:29.396368 10739 log.cc:824] T 00000000000000000000000000000000 P c47a8cee0bfb4f858d7bf08b5b105c2a: Log is configured to *not* fsync() on all Append() calls
I20260928 05:18:29.398800 10742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:29.379168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41420 (local address 127.0.97.62:46109)
0928 05:18:29.379901 (+ 733us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:29.379912 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:29.379951 (+ 39us) server_negotiation.cc:407] Connection header received
0928 05:18:29.381514 (+ 1563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:29.381534 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:29.381871 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:29.382163 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:18:29.383816 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:29.385277 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:29.387022 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:29.387657 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:29.387780 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:29.392218 (+ 4438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:18:29.392225 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:18:29.392239 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:18:29.392274 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:18:29.395142 (+ 2868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:18:29.395839 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:18:29.395846 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:18:29.395852 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:18:29.396012 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:18:29.396349 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:18:29.396353 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:18:29.396359 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:18:29.396893 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:18:29.397167 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0928 05:18:29.397550 (+ 383us) server_negotiation.cc:299] Negotiation successful
0928 05:18:29.397916 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":338,"threads_started":1}
I20260928 05:18:29.398999 10739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c47a8cee0bfb4f858d7bf08b5b105c2a: No bootstrap required, opened a new log
I20260928 05:18:29.406272 10739 raft_consensus.cc:348] T 00000000000000000000000000000000 P c47a8cee0bfb4f858d7bf08b5b105c2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c47a8cee0bfb4f858d7bf08b5b105c2a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46109 } }
I20260928 05:18:29.406584 10739 raft_consensus.cc:374] T 00000000000000000000000000000000 P c47a8cee0bfb4f858d7bf08b5b105c2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:18:29.406666 10739 raft_consensus.cc:729] T 00000000000000000000000000000000 P c47a8cee0bfb4f858d7bf08b5b105c2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c47a8cee0bfb4f858d7bf08b5b105c2a, State: Initialized, Role: FOLLOWER
I20260928 05:18:29.407150 10739 consensus_queue.cc:260] T 00000000000000000000000000000000 P c47a8cee0bfb4f858d7bf08b5b105c2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c47a8cee0bfb4f858d7bf08b5b105c2a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46109 } }
I20260928 05:18:29.407367 10739 raft_consensus.cc:388] T 00000000000000000000000000000000 P c47a8cee0bfb4f858d7bf08b5b105c2a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:18:29.407500 10739 raft_consensus.cc:482] T 00000000000000000000000000000000 P c47a8cee0bfb4f858d7bf08b5b105c2a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:18:29.407655 10739 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c47a8cee0bfb4f858d7bf08b5b105c2a [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:18:29.409575 10739 raft_consensus.cc:504] T 00000000000000000000000000000000 P c47a8cee0bfb4f858d7bf08b5b105c2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c47a8cee0bfb4f858d7bf08b5b105c2a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46109 } }
I20260928 05:18:29.409982 10739 leader_election.cc:302] T 00000000000000000000000000000000 P c47a8cee0bfb4f858d7bf08b5b105c2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c47a8cee0bfb4f858d7bf08b5b105c2a; no voters:
I20260928 05:18:29.410468 10739 leader_election.cc:288] T 00000000000000000000000000000000 P c47a8cee0bfb4f858d7bf08b5b105c2a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:18:29.410571 10744 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c47a8cee0bfb4f858d7bf08b5b105c2a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:18:29.411212 10744 raft_consensus.cc:686] T 00000000000000000000000000000000 P c47a8cee0bfb4f858d7bf08b5b105c2a [term 1 LEADER]: Becoming Leader. State: Replica: c47a8cee0bfb4f858d7bf08b5b105c2a, State: Running, Role: LEADER
I20260928 05:18:29.411725 10744 consensus_queue.cc:237] T 00000000000000000000000000000000 P c47a8cee0bfb4f858d7bf08b5b105c2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c47a8cee0bfb4f858d7bf08b5b105c2a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46109 } }
I20260928 05:18:29.412256 10739 sys_catalog.cc:564] T 00000000000000000000000000000000 P c47a8cee0bfb4f858d7bf08b5b105c2a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:18:29.414284 10745 sys_catalog.cc:455] T 00000000000000000000000000000000 P c47a8cee0bfb4f858d7bf08b5b105c2a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c47a8cee0bfb4f858d7bf08b5b105c2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c47a8cee0bfb4f858d7bf08b5b105c2a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46109 } } }
I20260928 05:18:29.414261 10746 sys_catalog.cc:455] T 00000000000000000000000000000000 P c47a8cee0bfb4f858d7bf08b5b105c2a [sys.catalog]: SysCatalogTable state changed. Reason: New leader c47a8cee0bfb4f858d7bf08b5b105c2a. Latest consensus state: current_term: 1 leader_uuid: "c47a8cee0bfb4f858d7bf08b5b105c2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c47a8cee0bfb4f858d7bf08b5b105c2a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46109 } } }
I20260928 05:18:29.414613 10745 sys_catalog.cc:458] T 00000000000000000000000000000000 P c47a8cee0bfb4f858d7bf08b5b105c2a [sys.catalog]: This master's current role is: LEADER
I20260928 05:18:29.414613 10746 sys_catalog.cc:458] T 00000000000000000000000000000000 P c47a8cee0bfb4f858d7bf08b5b105c2a [sys.catalog]: This master's current role is: LEADER
I20260928 05:18:29.416210 10752 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:18:29.420284 10752 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:18:29.428212 10752 catalog_manager.cc:1347] Generated new cluster ID: 7bcef98130a04a0595b0d2aff3fb27ff
I20260928 05:18:29.428304 10752 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:18:29.457418 10752 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:18:29.458590 10752 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:18:29.463826 10752 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c47a8cee0bfb4f858d7bf08b5b105c2a: Generated new TSK 0
I20260928 05:18:29.464588 10752 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572622576160-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572622576160-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260928 05:18:29.515192 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572622576160-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572622576160-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572622576160-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572622576160-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46109
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572622576160-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:18:29.808142 10765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:18:29.808377 10765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:18:29.808482 10765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:18:29.815490 10765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:18:29.815582 10765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:18:29.815692 10765 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 05:18:29.824762 10765 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572622576160-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572622576160-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572622576160-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572622576160-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46109
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572622576160-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:18:29.825914 10765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:18:29.827071 10765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:18:29.838601 10770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:18:29.838634 10771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:18:29.838600 10773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:18:29.839480 10765 server_base.cc:1034] running on GCE node
I20260928 05:18:29.840004 10765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:18:29.840775 10765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:18:29.841965 10765 hybrid_clock.cc:648] HybridClock initialized: now 1790572709841924 us; error 34 us; skew 500 ppm
I20260928 05:18:29.842253 10765 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:18:29 dist-test-slave-whsn krb5kdc[10659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572709, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:18:29.845109 10765 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260928 05:18:29.846508 10765 webserver.cc:460] Webserver started at http://127.0.97.1:44187/ using document root <none> and password file <none>
I20260928 05:18:29.847105 10765 fs_manager.cc:359] Metadata directory not provided
I20260928 05:18:29.847200 10765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:18:29.847462 10765 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:18:29.850088 10765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572622576160-388-0/minicluster-data/ts-0/data/instance:
uuid: "bf15c5852ece46dd9f2f6759ea6f254f"
format_stamp: "Formatted at 2026-09-28 05:18:29 on dist-test-slave-whsn"
server_key: "b2b02f6e11c3f76178d8becf857cc404"
server_key_iv: "9735a2724e558d6f9dee6eb665f38442"
server_key_version: "encryptionkey@0"
I20260928 05:18:29.850744 10765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572622576160-388-0/minicluster-data/ts-0/wal/instance:
uuid: "bf15c5852ece46dd9f2f6759ea6f254f"
format_stamp: "Formatted at 2026-09-28 05:18:29 on dist-test-slave-whsn"
server_key: "b2b02f6e11c3f76178d8becf857cc404"
server_key_iv: "9735a2724e558d6f9dee6eb665f38442"
server_key_version: "encryptionkey@0"
I20260928 05:18:29.856307 10765 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 05:18:29.860932 10780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:18:29.862402 10765 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260928 05:18:29.862536 10765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572622576160-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572622576160-388-0/minicluster-data/ts-0/wal
uuid: "bf15c5852ece46dd9f2f6759ea6f254f"
format_stamp: "Formatted at 2026-09-28 05:18:29 on dist-test-slave-whsn"
server_key: "b2b02f6e11c3f76178d8becf857cc404"
server_key_iv: "9735a2724e558d6f9dee6eb665f38442"
server_key_version: "encryptionkey@0"
I20260928 05:18:29.862882 10765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572622576160-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572622576160-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572622576160-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:18:29.886966 10765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:18:29.892004 10765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:18:29.892298 10765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:18:29.893478 10765 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:18:29.895211 10765 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:18:29.895299 10765 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:18:29.895421 10765 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:18:29.895483 10765 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:18:29.924137 10765 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37153
I20260928 05:18:29.924176 10892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37153 every 8 connection(s)
I20260928 05:18:29.925318 10765 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572622576160-388-0/minicluster-data/ts-0/data/info.pb
I20260928 05:18:29.930174 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 10765
I20260928 05:18:29.930337 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572622576160-388-0/minicluster-data/ts-0/wal/instance
I20260928 05:18:29.930960 388 external_mini_cluster.cc:1428] Setting key 989a05443be9dd4b52f294e5af56ee2e
Sep 28 05:18:29 dist-test-slave-whsn krb5kdc[10659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572709, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:18:29.942957 10895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:29.927493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43101 (local address 127.0.97.62:46109)
0928 05:18:29.927849 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:29.927853 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:29.928717 (+ 864us) server_negotiation.cc:407] Connection header received
0928 05:18:29.930557 (+ 1840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:29.930562 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:29.930635 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:29.930723 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:18:29.932218 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:29.932972 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:29.934385 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:29.934763 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:29.934862 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:29.937946 (+ 3084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:18:29.937951 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:18:29.937954 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:18:29.937988 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:18:29.940368 (+ 2380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:18:29.940989 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:18:29.940994 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:18:29.940998 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:18:29.941107 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:18:29.941474 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:18:29.941478 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:18:29.941480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:18:29.941717 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:18:29.941835 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 05:18:29.942420 (+ 585us) server_negotiation.cc:299] Negotiation successful
0928 05:18:29.942617 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":186,"threads_started":1}
I20260928 05:18:29.943781 10896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:29.927947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46109 (local address 127.0.97.1:43101)
0928 05:18:29.928580 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0928 05:18:29.928619 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 05:18:29.930313 (+ 1694us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:18:29.930974 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:18:29.930986 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:18:29.931473 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:18:29.932076 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:29.932092 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:29.933107 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:29.933111 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:29.934265 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:29.934275 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:29.934972 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:29.934980 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:29.935171 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:29.935731 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:18:29.935756 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:18:29.937750 (+ 1994us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:18:29.940484 (+ 2734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:18:29.940492 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:18:29.940507 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:18:29.940892 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:18:29.941198 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:18:29.941202 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:18:29.941206 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:18:29.941369 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:18:29.941825 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:18:29.941832 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:18:29.942112 (+ 280us) client_negotiation.cc:769] Sending connection context
0928 05:18:29.942321 (+ 209us) client_negotiation.cc:240] Negotiation successful
0928 05:18:29.942775 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":243,"threads_started":1}
I20260928 05:18:29.945232 10893 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46109
I20260928 05:18:29.945506 10893 heartbeater.cc:464] Registering TS with master...
I20260928 05:18:29.946170 10893 heartbeater.cc:511] Master 127.0.97.62:46109 requested a full tablet report, sending...
I20260928 05:18:29.948355 10704 ts_manager.cc:194] Registered new tserver with Master: bf15c5852ece46dd9f2f6759ea6f254f (127.0.97.1:37153)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260928 05:18:29.950361 10704 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:43101
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572622576160-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572622576160-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260928 05:18:29.973338 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572622576160-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572622576160-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572622576160-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572622576160-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46109
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572622576160-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:18:30.266527 10899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:18:30.266765 10899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:18:30.266860 10899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:18:30.273592 10899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:18:30.273679 10899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:18:30.273785 10899 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 05:18:30.282521 10899 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572622576160-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572622576160-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572622576160-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572622576160-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46109
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572622576160-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:18:30.283550 10899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:18:30.284679 10899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:18:30.296154 10904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:18:30.296121 10905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:18:30.296180 10907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:18:30.296801 10899 server_base.cc:1034] running on GCE node
I20260928 05:18:30.297345 10899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:18:30.298164 10899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:18:30.299353 10899 hybrid_clock.cc:648] HybridClock initialized: now 1790572710299300 us; error 43 us; skew 500 ppm
I20260928 05:18:30.299629 10899 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:18:30 dist-test-slave-whsn krb5kdc[10659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572710, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:18:30.306717 10899 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260928 05:18:30.307921 10899 webserver.cc:460] Webserver started at http://127.0.97.2:36005/ using document root <none> and password file <none>
I20260928 05:18:30.308539 10899 fs_manager.cc:359] Metadata directory not provided
I20260928 05:18:30.308658 10899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:18:30.308931 10899 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:18:30.311501 10899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572622576160-388-0/minicluster-data/ts-1/data/instance:
uuid: "949ca59be90f4906b1c2f632fbb3d9fb"
format_stamp: "Formatted at 2026-09-28 05:18:30 on dist-test-slave-whsn"
server_key: "d28cc3dc8ef966eebb140266a78afd83"
server_key_iv: "f96be4870de342923c6f887d527ea67d"
server_key_version: "encryptionkey@0"
I20260928 05:18:30.312178 10899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572622576160-388-0/minicluster-data/ts-1/wal/instance:
uuid: "949ca59be90f4906b1c2f632fbb3d9fb"
format_stamp: "Formatted at 2026-09-28 05:18:30 on dist-test-slave-whsn"
server_key: "d28cc3dc8ef966eebb140266a78afd83"
server_key_iv: "f96be4870de342923c6f887d527ea67d"
server_key_version: "encryptionkey@0"
I20260928 05:18:30.316658 10899 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260928 05:18:30.319687 10914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:18:30.320833 10899 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 05:18:30.320995 10899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572622576160-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572622576160-388-0/minicluster-data/ts-1/wal
uuid: "949ca59be90f4906b1c2f632fbb3d9fb"
format_stamp: "Formatted at 2026-09-28 05:18:30 on dist-test-slave-whsn"
server_key: "d28cc3dc8ef966eebb140266a78afd83"
server_key_iv: "f96be4870de342923c6f887d527ea67d"
server_key_version: "encryptionkey@0"
I20260928 05:18:30.321146 10899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572622576160-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572622576160-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572622576160-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:18:30.335448 10899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:18:30.338640 10899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:18:30.338861 10899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:18:30.339628 10899 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:18:30.340739 10899 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:18:30.340809 10899 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:18:30.340941 10899 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:18:30.340992 10899 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:18:30.368839 10899 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42133
I20260928 05:18:30.368909 11026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42133 every 8 connection(s)
I20260928 05:18:30.369983 10899 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572622576160-388-0/minicluster-data/ts-1/data/info.pb
I20260928 05:18:30.379683 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 10899
I20260928 05:18:30.379844 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572622576160-388-0/minicluster-data/ts-1/wal/instance
I20260928 05:18:30.380257 388 external_mini_cluster.cc:1428] Setting key f8a6e9f6a4d34cc4913e284c8da0d7a9
Sep 28 05:18:30 dist-test-slave-whsn krb5kdc[10659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572710, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:18:30.391055 10895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:30.372129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49391 (local address 127.0.97.62:46109)
0928 05:18:30.372281 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:30.372285 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:30.373375 (+ 1090us) server_negotiation.cc:407] Connection header received
0928 05:18:30.375471 (+ 2096us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:30.375475 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:30.375538 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:30.375619 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:18:30.377368 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:30.378067 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:30.380228 (+ 2161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:30.380799 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:30.380938 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:30.384835 (+ 3897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:18:30.384841 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:18:30.384845 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:18:30.384907 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:18:30.387964 (+ 3057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:18:30.388745 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:18:30.388750 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:18:30.388753 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:18:30.388931 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:18:30.389331 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:18:30.389336 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:18:30.389339 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:18:30.389651 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:18:30.389774 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 05:18:30.390529 (+ 755us) server_negotiation.cc:299] Negotiation successful
0928 05:18:30.390732 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 05:18:30.391893 11029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:30.372577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46109 (local address 127.0.97.2:49391)
0928 05:18:30.373239 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0928 05:18:30.373273 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 05:18:30.375210 (+ 1937us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:18:30.375863 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:18:30.375874 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:18:30.376521 (+ 647us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:18:30.377210 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:30.377231 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:30.378203 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:30.378208 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:30.380089 (+ 1881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:30.380103 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:30.381047 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:30.381054 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:30.381274 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:30.381935 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:18:30.381968 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:18:30.384619 (+ 2651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:18:30.388088 (+ 3469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:18:30.388096 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:18:30.388110 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:18:30.388633 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:18:30.389018 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:18:30.389023 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:18:30.389026 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:18:30.389224 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:18:30.389773 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:18:30.389781 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:18:30.390155 (+ 374us) client_negotiation.cc:769] Sending connection context
0928 05:18:30.390411 (+ 256us) client_negotiation.cc:240] Negotiation successful
0928 05:18:30.390856 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":277,"threads_started":1}
I20260928 05:18:30.393115 11027 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46109
I20260928 05:18:30.393392 11027 heartbeater.cc:464] Registering TS with master...
I20260928 05:18:30.394131 11027 heartbeater.cc:511] Master 127.0.97.62:46109 requested a full tablet report, sending...
I20260928 05:18:30.395447 10704 ts_manager.cc:194] Registered new tserver with Master: 949ca59be90f4906b1c2f632fbb3d9fb (127.0.97.2:42133)
I20260928 05:18:30.396183 10704 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:49391
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572622576160-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572622576160-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260928 05:18:30.425082 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572622576160-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572622576160-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572622576160-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572622576160-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46109
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572622576160-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:18:30.719877 11032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:18:30.720103 11032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:18:30.720193 11032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:18:30.726995 11032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:18:30.727080 11032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:18:30.727181 11032 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 05:18:30.735961 11032 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572622576160-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572622576160-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572622576160-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572622576160-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46109
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572622576160-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:18:30.737025 11032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:18:30.738148 11032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:18:30.751698 11037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:18:30.751724 11040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:18:30.751802 11038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:18:30.752669 11032 server_base.cc:1034] running on GCE node
I20260928 05:18:30.753230 11032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:18:30.754009 11032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:18:30.755196 11032 hybrid_clock.cc:648] HybridClock initialized: now 1790572710755156 us; error 39 us; skew 500 ppm
I20260928 05:18:30.755467 11032 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:18:30 dist-test-slave-whsn krb5kdc[10659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572710, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:18:30.758164 11032 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260928 05:18:30.759415 11032 webserver.cc:460] Webserver started at http://127.0.97.3:46313/ using document root <none> and password file <none>
I20260928 05:18:30.760008 11032 fs_manager.cc:359] Metadata directory not provided
I20260928 05:18:30.760113 11032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:18:30.760356 11032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:18:30.762948 11032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572622576160-388-0/minicluster-data/ts-2/data/instance:
uuid: "ecf559638fc047db83f9411c8fa031c6"
format_stamp: "Formatted at 2026-09-28 05:18:30 on dist-test-slave-whsn"
server_key: "4abc3b20b140625ed54c38bbc12451f8"
server_key_iv: "c3d1a765a1963cf630a44eda169a0b44"
server_key_version: "encryptionkey@0"
I20260928 05:18:30.763600 11032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572622576160-388-0/minicluster-data/ts-2/wal/instance:
uuid: "ecf559638fc047db83f9411c8fa031c6"
format_stamp: "Formatted at 2026-09-28 05:18:30 on dist-test-slave-whsn"
server_key: "4abc3b20b140625ed54c38bbc12451f8"
server_key_iv: "c3d1a765a1963cf630a44eda169a0b44"
server_key_version: "encryptionkey@0"
I20260928 05:18:30.767995 11032 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 05:18:30.771008 11047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:18:30.772167 11032 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 05:18:30.772300 11032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572622576160-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572622576160-388-0/minicluster-data/ts-2/wal
uuid: "ecf559638fc047db83f9411c8fa031c6"
format_stamp: "Formatted at 2026-09-28 05:18:30 on dist-test-slave-whsn"
server_key: "4abc3b20b140625ed54c38bbc12451f8"
server_key_iv: "c3d1a765a1963cf630a44eda169a0b44"
server_key_version: "encryptionkey@0"
I20260928 05:18:30.772469 11032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572622576160-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572622576160-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572622576160-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:18:30.799803 11032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:18:30.803259 11032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:18:30.803504 11032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:18:30.804337 11032 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:18:30.805637 11032 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:18:30.805707 11032 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:18:30.805819 11032 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:18:30.805877 11032 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:18:30.834980 11032 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39287
I20260928 05:18:30.835042 11159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39287 every 8 connection(s)
I20260928 05:18:30.836103 11032 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572622576160-388-0/minicluster-data/ts-2/data/info.pb
I20260928 05:18:30.841338 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 11032
I20260928 05:18:30.841511 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572622576160-388-0/minicluster-data/ts-2/wal/instance
I20260928 05:18:30.842012 388 external_mini_cluster.cc:1428] Setting key 6096110a9b6a4874ff661291eb0e7bd2
Sep 28 05:18:30 dist-test-slave-whsn krb5kdc[10659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572710, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:18:30.854151 10895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:30.838442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52087 (local address 127.0.97.62:46109)
0928 05:18:30.838629 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:30.838635 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:30.839718 (+ 1083us) server_negotiation.cc:407] Connection header received
0928 05:18:30.841597 (+ 1879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:30.841603 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:30.841669 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:30.841748 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:18:30.843493 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:30.844470 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:30.846037 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:30.846415 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:30.846508 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:30.849559 (+ 3051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:18:30.849564 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:18:30.849567 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:18:30.849596 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:18:30.851881 (+ 2285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:18:30.852463 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:18:30.852467 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:18:30.852469 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:18:30.852565 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:18:30.852898 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:18:30.852901 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:18:30.852903 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:18:30.853138 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:18:30.853238 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 05:18:30.853786 (+ 548us) server_negotiation.cc:299] Negotiation successful
0928 05:18:30.853921 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 05:18:30.855057 11162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:30.838902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46109 (local address 127.0.97.3:52087)
0928 05:18:30.839566 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0928 05:18:30.839607 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 05:18:30.841299 (+ 1692us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:18:30.842024 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:18:30.842034 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:18:30.842633 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:18:30.843337 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:30.843362 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:30.844619 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:30.844622 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:30.845931 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:30.845943 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:30.846575 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:30.846580 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:30.846761 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:30.847320 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:18:30.847340 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:18:30.849380 (+ 2040us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:18:30.852003 (+ 2623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:18:30.852009 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:18:30.852020 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:18:30.852369 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:18:30.852633 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:18:30.852636 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:18:30.852638 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:18:30.852791 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:18:30.853225 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:18:30.853230 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:18:30.853508 (+ 278us) client_negotiation.cc:769] Sending connection context
0928 05:18:30.853708 (+ 200us) client_negotiation.cc:240] Negotiation successful
0928 05:18:30.854057 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":270,"threads_started":1}
I20260928 05:18:30.856348 11160 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46109
I20260928 05:18:30.856645 11160 heartbeater.cc:464] Registering TS with master...
I20260928 05:18:30.857292 11160 heartbeater.cc:511] Master 127.0.97.62:46109 requested a full tablet report, sending...
I20260928 05:18:30.858480 10704 ts_manager.cc:194] Registered new tserver with Master: ecf559638fc047db83f9411c8fa031c6 (127.0.97.3:39287)
I20260928 05:18:30.859180 10704 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:52087
I20260928 05:18:30.868641 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 05:18:30.883715 10895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:30.872704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41434 (local address 127.0.97.62:46109)
0928 05:18:30.872841 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:30.872844 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:30.873316 (+ 472us) server_negotiation.cc:407] Connection header received
0928 05:18:30.873439 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:30.873442 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:30.873504 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:30.873576 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:18:30.874818 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:30.875545 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:30.876790 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:30.877217 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:30.877303 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:30.879361 (+ 2058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:18:30.879365 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:18:30.879368 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:18:30.879394 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:18:30.881668 (+ 2274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:18:30.882292 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:18:30.882296 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:18:30.882298 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:18:30.882403 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:18:30.882660 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:18:30.882663 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:18:30.882664 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:18:30.882914 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:18:30.883087 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0928 05:18:30.883363 (+ 276us) server_negotiation.cc:299] Negotiation successful
0928 05:18:30.883502 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 05:18:30.887152 10704 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41434:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 05:18:30.893543 10704 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 05:18:30.925007 11176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:30.904389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37153 (local address 127.0.0.1:36724)
0928 05:18:30.917012 (+ 12623us) negotiation.cc:107] Waiting for socket to connect
0928 05:18:30.917028 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 05:18:30.917132 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:18:30.918725 (+ 1593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:18:30.918728 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:18:30.918748 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:18:30.919020 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:30.919029 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:30.920646 (+ 1617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:30.920654 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:30.922489 (+ 1835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:30.922500 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:30.924027 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:30.924033 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:30.924206 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:30.924264 (+ 58us) client_negotiation.cc:769] Sending connection context
0928 05:18:30.924363 (+ 99us) client_negotiation.cc:240] Negotiation successful
0928 05:18:30.924536 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12509,"mutex_wait_us":77,"thread_start_us":11281,"threads_started":1}
I20260928 05:18:30.925740 11174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:30.904138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42133 (local address 127.0.0.1:35172)
0928 05:18:30.915231 (+ 11093us) negotiation.cc:107] Waiting for socket to connect
0928 05:18:30.915285 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 05:18:30.915497 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:18:30.917541 (+ 2044us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:18:30.917548 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:18:30.917581 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:18:30.918045 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:30.918059 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:30.920505 (+ 2446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:30.920509 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:30.922939 (+ 2430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:30.922953 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:30.925019 (+ 2066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:30.925026 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:30.925176 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:30.925204 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 05:18:30.925277 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 05:18:30.925382 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10804,"mutex_wait_us":82,"thread_start_us":13645,"threads_started":1}
I20260928 05:18:30.926473 11173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:30.904200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35172 (local address 127.0.97.2:42133)
0928 05:18:30.916661 (+ 12461us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:30.916670 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:30.916702 (+ 32us) server_negotiation.cc:407] Connection header received
0928 05:18:30.916807 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:30.916815 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:30.917102 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:30.917288 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:18:30.918963 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:30.920338 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:30.923092 (+ 2754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:30.924490 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:30.924597 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:30.924793 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0928 05:18:30.926032 (+ 1239us) server_negotiation.cc:299] Negotiation successful
0928 05:18:30.926166 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12336,"thread_start_us":12245,"threads_started":1}
I20260928 05:18:30.926496 11175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:30.904953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36724 (local address 127.0.97.1:37153)
0928 05:18:30.918265 (+ 13312us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:30.918272 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:30.918292 (+ 20us) server_negotiation.cc:407] Connection header received
0928 05:18:30.918393 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:30.918399 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:30.918570 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:30.918742 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:18:30.919185 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:30.920502 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:30.922620 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:30.923800 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:30.925714 (+ 1914us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:30.925933 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0928 05:18:30.926007 (+ 74us) server_negotiation.cc:299] Negotiation successful
0928 05:18:30.926165 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13180,"thread_start_us":13179,"threads_started":1}
I20260928 05:18:30.926616 11172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:30.904878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39287 (local address 127.0.0.1:40286)
0928 05:18:30.917566 (+ 12688us) negotiation.cc:107] Waiting for socket to connect
0928 05:18:30.917580 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 05:18:30.917678 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:18:30.919029 (+ 1351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:18:30.919034 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:18:30.919058 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:18:30.919423 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:30.919435 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:30.921246 (+ 1811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:30.921251 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:30.923346 (+ 2095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:30.923360 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:30.925929 (+ 2569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:30.925935 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:30.926085 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:30.926116 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 05:18:30.926177 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 05:18:30.926290 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12573,"mutex_wait_us":26,"thread_start_us":11889,"threads_started":1}
I20260928 05:18:30.928408 11177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:30.905420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40286 (local address 127.0.97.3:39287)
0928 05:18:30.918223 (+ 12803us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:30.918231 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:30.918260 (+ 29us) server_negotiation.cc:407] Connection header received
0928 05:18:30.918347 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:30.918353 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:30.918562 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:30.918742 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:18:30.919593 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:30.921093 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:30.924257 (+ 3164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:30.925728 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:30.926733 (+ 1005us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:30.926928 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0928 05:18:30.926989 (+ 61us) server_negotiation.cc:299] Negotiation successful
0928 05:18:30.928127 (+ 1138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12631,"thread_start_us":12499,"threads_started":1}
I20260928 05:18:30.930188 10828 tablet_service.cc:1467] Processing CreateTablet for tablet 1fa9c6ee0cd9450fb11b479dd951546a (DEFAULT_TABLE table=test-table [id=8cbecdbd62e54d5f9c9aab696536a7c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:18:30.931622 10828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fa9c6ee0cd9450fb11b479dd951546a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:18:30.931303 10962 tablet_service.cc:1467] Processing CreateTablet for tablet 1fa9c6ee0cd9450fb11b479dd951546a (DEFAULT_TABLE table=test-table [id=8cbecdbd62e54d5f9c9aab696536a7c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:18:30.931468 11095 tablet_service.cc:1467] Processing CreateTablet for tablet 1fa9c6ee0cd9450fb11b479dd951546a (DEFAULT_TABLE table=test-table [id=8cbecdbd62e54d5f9c9aab696536a7c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:18:30.932487 10962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fa9c6ee0cd9450fb11b479dd951546a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:18:30.932665 11095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fa9c6ee0cd9450fb11b479dd951546a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:18:30.942766 11178 tablet_bootstrap.cc:492] T 1fa9c6ee0cd9450fb11b479dd951546a P ecf559638fc047db83f9411c8fa031c6: Bootstrap starting.
I20260928 05:18:30.944918 11179 tablet_bootstrap.cc:492] T 1fa9c6ee0cd9450fb11b479dd951546a P bf15c5852ece46dd9f2f6759ea6f254f: Bootstrap starting.
I20260928 05:18:30.947643 11180 tablet_bootstrap.cc:492] T 1fa9c6ee0cd9450fb11b479dd951546a P 949ca59be90f4906b1c2f632fbb3d9fb: Bootstrap starting.
I20260928 05:18:30.947659 11178 tablet_bootstrap.cc:654] T 1fa9c6ee0cd9450fb11b479dd951546a P ecf559638fc047db83f9411c8fa031c6: Neither blocks nor log segments found. Creating new log.
I20260928 05:18:30.948246 11179 tablet_bootstrap.cc:654] T 1fa9c6ee0cd9450fb11b479dd951546a P bf15c5852ece46dd9f2f6759ea6f254f: Neither blocks nor log segments found. Creating new log.
I20260928 05:18:30.949033 11178 log.cc:824] T 1fa9c6ee0cd9450fb11b479dd951546a P ecf559638fc047db83f9411c8fa031c6: Log is configured to *not* fsync() on all Append() calls
I20260928 05:18:30.949319 11179 log.cc:824] T 1fa9c6ee0cd9450fb11b479dd951546a P bf15c5852ece46dd9f2f6759ea6f254f: Log is configured to *not* fsync() on all Append() calls
I20260928 05:18:30.950309 11180 tablet_bootstrap.cc:654] T 1fa9c6ee0cd9450fb11b479dd951546a P 949ca59be90f4906b1c2f632fbb3d9fb: Neither blocks nor log segments found. Creating new log.
I20260928 05:18:30.951153 11180 log.cc:824] T 1fa9c6ee0cd9450fb11b479dd951546a P 949ca59be90f4906b1c2f632fbb3d9fb: Log is configured to *not* fsync() on all Append() calls
I20260928 05:18:30.951736 11179 tablet_bootstrap.cc:492] T 1fa9c6ee0cd9450fb11b479dd951546a P bf15c5852ece46dd9f2f6759ea6f254f: No bootstrap required, opened a new log
I20260928 05:18:30.951978 11179 ts_tablet_manager.cc:1397] T 1fa9c6ee0cd9450fb11b479dd951546a P bf15c5852ece46dd9f2f6759ea6f254f: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260928 05:18:30.953013 10893 heartbeater.cc:503] Master 127.0.97.62:46109 was elected leader, sending a full tablet report...
I20260928 05:18:30.953527 11180 tablet_bootstrap.cc:492] T 1fa9c6ee0cd9450fb11b479dd951546a P 949ca59be90f4906b1c2f632fbb3d9fb: No bootstrap required, opened a new log
I20260928 05:18:30.953743 11180 ts_tablet_manager.cc:1397] T 1fa9c6ee0cd9450fb11b479dd951546a P 949ca59be90f4906b1c2f632fbb3d9fb: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260928 05:18:30.958339 11178 tablet_bootstrap.cc:492] T 1fa9c6ee0cd9450fb11b479dd951546a P ecf559638fc047db83f9411c8fa031c6: No bootstrap required, opened a new log
I20260928 05:18:30.958581 11178 ts_tablet_manager.cc:1397] T 1fa9c6ee0cd9450fb11b479dd951546a P ecf559638fc047db83f9411c8fa031c6: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.000s
I20260928 05:18:30.959800 11179 raft_consensus.cc:348] T 1fa9c6ee0cd9450fb11b479dd951546a P bf15c5852ece46dd9f2f6759ea6f254f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "949ca59be90f4906b1c2f632fbb3d9fb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42133 } } peers { permanent_uuid: "bf15c5852ece46dd9f2f6759ea6f254f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37153 } } peers { permanent_uuid: "ecf559638fc047db83f9411c8fa031c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39287 } }
I20260928 05:18:30.960088 11179 raft_consensus.cc:374] T 1fa9c6ee0cd9450fb11b479dd951546a P bf15c5852ece46dd9f2f6759ea6f254f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:18:30.960177 11179 raft_consensus.cc:729] T 1fa9c6ee0cd9450fb11b479dd951546a P bf15c5852ece46dd9f2f6759ea6f254f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf15c5852ece46dd9f2f6759ea6f254f, State: Initialized, Role: FOLLOWER
I20260928 05:18:30.960666 11179 consensus_queue.cc:260] T 1fa9c6ee0cd9450fb11b479dd951546a P bf15c5852ece46dd9f2f6759ea6f254f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "949ca59be90f4906b1c2f632fbb3d9fb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42133 } } peers { permanent_uuid: "bf15c5852ece46dd9f2f6759ea6f254f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37153 } } peers { permanent_uuid: "ecf559638fc047db83f9411c8fa031c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39287 } }
I20260928 05:18:30.962086 11180 raft_consensus.cc:348] T 1fa9c6ee0cd9450fb11b479dd951546a P 949ca59be90f4906b1c2f632fbb3d9fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "949ca59be90f4906b1c2f632fbb3d9fb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42133 } } peers { permanent_uuid: "bf15c5852ece46dd9f2f6759ea6f254f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37153 } } peers { permanent_uuid: "ecf559638fc047db83f9411c8fa031c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39287 } }
I20260928 05:18:30.962479 11180 raft_consensus.cc:374] T 1fa9c6ee0cd9450fb11b479dd951546a P 949ca59be90f4906b1c2f632fbb3d9fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:18:30.962576 11180 raft_consensus.cc:729] T 1fa9c6ee0cd9450fb11b479dd951546a P 949ca59be90f4906b1c2f632fbb3d9fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 949ca59be90f4906b1c2f632fbb3d9fb, State: Initialized, Role: FOLLOWER
I20260928 05:18:30.963078 11180 consensus_queue.cc:260] T 1fa9c6ee0cd9450fb11b479dd951546a P 949ca59be90f4906b1c2f632fbb3d9fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "949ca59be90f4906b1c2f632fbb3d9fb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42133 } } peers { permanent_uuid: "bf15c5852ece46dd9f2f6759ea6f254f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37153 } } peers { permanent_uuid: "ecf559638fc047db83f9411c8fa031c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39287 } }
I20260928 05:18:30.964666 11027 heartbeater.cc:503] Master 127.0.97.62:46109 was elected leader, sending a full tablet report...
I20260928 05:18:30.965246 11180 ts_tablet_manager.cc:1428] T 1fa9c6ee0cd9450fb11b479dd951546a P 949ca59be90f4906b1c2f632fbb3d9fb: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260928 05:18:30.967818 11178 raft_consensus.cc:348] T 1fa9c6ee0cd9450fb11b479dd951546a P ecf559638fc047db83f9411c8fa031c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "949ca59be90f4906b1c2f632fbb3d9fb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42133 } } peers { permanent_uuid: "bf15c5852ece46dd9f2f6759ea6f254f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37153 } } peers { permanent_uuid: "ecf559638fc047db83f9411c8fa031c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39287 } }
I20260928 05:18:30.968180 11178 raft_consensus.cc:374] T 1fa9c6ee0cd9450fb11b479dd951546a P ecf559638fc047db83f9411c8fa031c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:18:30.968276 11178 raft_consensus.cc:729] T 1fa9c6ee0cd9450fb11b479dd951546a P ecf559638fc047db83f9411c8fa031c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecf559638fc047db83f9411c8fa031c6, State: Initialized, Role: FOLLOWER
I20260928 05:18:30.968343 11179 ts_tablet_manager.cc:1428] T 1fa9c6ee0cd9450fb11b479dd951546a P bf15c5852ece46dd9f2f6759ea6f254f: Time spent starting tablet: real 0.016s user 0.013s sys 0.004s
I20260928 05:18:30.968801 11178 consensus_queue.cc:260] T 1fa9c6ee0cd9450fb11b479dd951546a P ecf559638fc047db83f9411c8fa031c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "949ca59be90f4906b1c2f632fbb3d9fb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42133 } } peers { permanent_uuid: "bf15c5852ece46dd9f2f6759ea6f254f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37153 } } peers { permanent_uuid: "ecf559638fc047db83f9411c8fa031c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39287 } }
I20260928 05:18:30.969861 11160 heartbeater.cc:503] Master 127.0.97.62:46109 was elected leader, sending a full tablet report...
I20260928 05:18:30.970530 11178 ts_tablet_manager.cc:1428] T 1fa9c6ee0cd9450fb11b479dd951546a P ecf559638fc047db83f9411c8fa031c6: Time spent starting tablet: real 0.012s user 0.009s sys 0.004s
W20260928 05:18:31.087623 11161 tablet.cc:2369] T 1fa9c6ee0cd9450fb11b479dd951546a P ecf559638fc047db83f9411c8fa031c6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 05:18:31.121681 11028 tablet.cc:2369] T 1fa9c6ee0cd9450fb11b479dd951546a P 949ca59be90f4906b1c2f632fbb3d9fb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 05:18:31.177781 10894 tablet.cc:2369] T 1fa9c6ee0cd9450fb11b479dd951546a P bf15c5852ece46dd9f2f6759ea6f254f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 05:18:31.354799 11184 raft_consensus.cc:482] T 1fa9c6ee0cd9450fb11b479dd951546a P bf15c5852ece46dd9f2f6759ea6f254f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:18:31.355060 11184 raft_consensus.cc:504] T 1fa9c6ee0cd9450fb11b479dd951546a P bf15c5852ece46dd9f2f6759ea6f254f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "949ca59be90f4906b1c2f632fbb3d9fb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42133 } } peers { permanent_uuid: "bf15c5852ece46dd9f2f6759ea6f254f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37153 } } peers { permanent_uuid: "ecf559638fc047db83f9411c8fa031c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39287 } }
I20260928 05:18:31.356642 11184 leader_election.cc:288] T 1fa9c6ee0cd9450fb11b479dd951546a P bf15c5852ece46dd9f2f6759ea6f254f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 949ca59be90f4906b1c2f632fbb3d9fb (127.0.97.2:42133), ecf559638fc047db83f9411c8fa031c6 (127.0.97.3:39287)
I20260928 05:18:31.364269 11177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:31.357000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54201 (local address 127.0.97.3:39287)
0928 05:18:31.357165 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:31.357170 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:31.357793 (+ 623us) server_negotiation.cc:407] Connection header received
0928 05:18:31.358005 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:31.358010 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:31.358074 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:31.358162 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:18:31.358819 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:31.359846 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:31.362548 (+ 2702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:31.363376 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:31.363499 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:31.363589 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 05:18:31.363913 (+ 324us) server_negotiation.cc:299] Negotiation successful
0928 05:18:31.364035 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 05:18:31.364347 11188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:31.356907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39287 (local address 127.0.97.1:54201)
0928 05:18:31.357688 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0928 05:18:31.357707 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 05:18:31.357860 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:18:31.358185 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:18:31.358190 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:18:31.358274 (+ 84us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:18:31.358708 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:31.358718 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:31.359961 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:31.359966 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:31.362393 (+ 2427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:31.362408 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:31.363577 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:31.363584 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:31.363736 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:31.363794 (+ 58us) client_negotiation.cc:769] Sending connection context
0928 05:18:31.363862 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 05:18:31.363982 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"mutex_wait_us":143,"spinlock_wait_cycles":21760,"thread_start_us":513,"threads_started":1}
I20260928 05:18:31.364508 11173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:31.357108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52143 (local address 127.0.97.2:42133)
0928 05:18:31.357276 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:31.357281 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:31.357858 (+ 577us) server_negotiation.cc:407] Connection header received
0928 05:18:31.357986 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:31.357991 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:31.358067 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:31.358162 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:18:31.358791 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:31.359846 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:31.362590 (+ 2744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:31.363403 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:31.363536 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:31.363623 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 05:18:31.364151 (+ 528us) server_negotiation.cc:299] Negotiation successful
0928 05:18:31.364273 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 05:18:31.364691 11187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:31.356970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42133 (local address 127.0.97.1:52143)
0928 05:18:31.357772 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0928 05:18:31.357785 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 05:18:31.357872 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:18:31.358185 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:18:31.358190 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:18:31.358216 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:18:31.358634 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:31.358649 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:31.359958 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:31.359963 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:31.362393 (+ 2430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:18:31.362408 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:18:31.363739 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:18:31.363744 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:18:31.363987 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:31.364019 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 05:18:31.364113 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 05:18:31.364227 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":710,"mutex_wait_us":179,"thread_start_us":364,"threads_started":1}
I20260928 05:18:31.365466 11115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fa9c6ee0cd9450fb11b479dd951546a" candidate_uuid: "bf15c5852ece46dd9f2f6759ea6f254f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecf559638fc047db83f9411c8fa031c6" is_pre_election: true
I20260928 05:18:31.365947 11115 raft_consensus.cc:2454] T 1fa9c6ee0cd9450fb11b479dd951546a P ecf559638fc047db83f9411c8fa031c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf15c5852ece46dd9f2f6759ea6f254f in term 0.
I20260928 05:18:31.365965 10982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fa9c6ee0cd9450fb11b479dd951546a" candidate_uuid: "bf15c5852ece46dd9f2f6759ea6f254f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "949ca59be90f4906b1c2f632fbb3d9fb" is_pre_election: true
I20260928 05:18:31.366464 10982 raft_consensus.cc:2454] T 1fa9c6ee0cd9450fb11b479dd951546a P 949ca59be90f4906b1c2f632fbb3d9fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf15c5852ece46dd9f2f6759ea6f254f in term 0.
I20260928 05:18:31.366587 10782 leader_election.cc:302] T 1fa9c6ee0cd9450fb11b479dd951546a P bf15c5852ece46dd9f2f6759ea6f254f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bf15c5852ece46dd9f2f6759ea6f254f, ecf559638fc047db83f9411c8fa031c6; no voters:
I20260928 05:18:31.366979 11184 raft_consensus.cc:2781] T 1fa9c6ee0cd9450fb11b479dd951546a P bf15c5852ece46dd9f2f6759ea6f254f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 05:18:31.367125 11184 raft_consensus.cc:482] T 1fa9c6ee0cd9450fb11b479dd951546a P bf15c5852ece46dd9f2f6759ea6f254f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 05:18:31.367211 11184 raft_consensus.cc:3037] T 1fa9c6ee0cd9450fb11b479dd951546a P bf15c5852ece46dd9f2f6759ea6f254f [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:18:31.369560 11184 raft_consensus.cc:504] T 1fa9c6ee0cd9450fb11b479dd951546a P bf15c5852ece46dd9f2f6759ea6f254f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "949ca59be90f4906b1c2f632fbb3d9fb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42133 } } peers { permanent_uuid: "bf15c5852ece46dd9f2f6759ea6f254f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37153 } } peers { permanent_uuid: "ecf559638fc047db83f9411c8fa031c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39287 } }
I20260928 05:18:31.370071 11184 leader_election.cc:288] T 1fa9c6ee0cd9450fb11b479dd951546a P bf15c5852ece46dd9f2f6759ea6f254f [CANDIDATE]: Term 1 election: Requested vote from peers 949ca59be90f4906b1c2f632fbb3d9fb (127.0.97.2:42133), ecf559638fc047db83f9411c8fa031c6 (127.0.97.3:39287)
I20260928 05:18:31.370441 10982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fa9c6ee0cd9450fb11b479dd951546a" candidate_uuid: "bf15c5852ece46dd9f2f6759ea6f254f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "949ca59be90f4906b1c2f632fbb3d9fb"
I20260928 05:18:31.370489 11115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fa9c6ee0cd9450fb11b479dd951546a" candidate_uuid: "bf15c5852ece46dd9f2f6759ea6f254f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecf559638fc047db83f9411c8fa031c6"
I20260928 05:18:31.370631 10982 raft_consensus.cc:3037] T 1fa9c6ee0cd9450fb11b479dd951546a P 949ca59be90f4906b1c2f632fbb3d9fb [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:18:31.370651 11115 raft_consensus.cc:3037] T 1fa9c6ee0cd9450fb11b479dd951546a P ecf559638fc047db83f9411c8fa031c6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:18:31.372645 11115 raft_consensus.cc:2454] T 1fa9c6ee0cd9450fb11b479dd951546a P ecf559638fc047db83f9411c8fa031c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf15c5852ece46dd9f2f6759ea6f254f in term 1.
I20260928 05:18:31.372972 10782 leader_election.cc:302] T 1fa9c6ee0cd9450fb11b479dd951546a P bf15c5852ece46dd9f2f6759ea6f254f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bf15c5852ece46dd9f2f6759ea6f254f, ecf559638fc047db83f9411c8fa031c6; no voters:
I20260928 05:18:31.373201 11184 raft_consensus.cc:2781] T 1fa9c6ee0cd9450fb11b479dd951546a P bf15c5852ece46dd9f2f6759ea6f254f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:18:31.373624 11184 raft_consensus.cc:686] T 1fa9c6ee0cd9450fb11b479dd951546a P bf15c5852ece46dd9f2f6759ea6f254f [term 1 LEADER]: Becoming Leader. State: Replica: bf15c5852ece46dd9f2f6759ea6f254f, State: Running, Role: LEADER
I20260928 05:18:31.373813 10982 raft_consensus.cc:2454] T 1fa9c6ee0cd9450fb11b479dd951546a P 949ca59be90f4906b1c2f632fbb3d9fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf15c5852ece46dd9f2f6759ea6f254f in term 1.
I20260928 05:18:31.374107 11184 consensus_queue.cc:237] T 1fa9c6ee0cd9450fb11b479dd951546a P bf15c5852ece46dd9f2f6759ea6f254f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "949ca59be90f4906b1c2f632fbb3d9fb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42133 } } peers { permanent_uuid: "bf15c5852ece46dd9f2f6759ea6f254f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37153 } } peers { permanent_uuid: "ecf559638fc047db83f9411c8fa031c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39287 } }
I20260928 05:18:31.381699 10703 catalog_manager.cc:5495] T 1fa9c6ee0cd9450fb11b479dd951546a P bf15c5852ece46dd9f2f6759ea6f254f reported cstate change: term changed from 0 to 1, leader changed from <none> to bf15c5852ece46dd9f2f6759ea6f254f (127.0.97.1). New cstate: current_term: 1 leader_uuid: "bf15c5852ece46dd9f2f6759ea6f254f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "949ca59be90f4906b1c2f632fbb3d9fb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf15c5852ece46dd9f2f6759ea6f254f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37153 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecf559638fc047db83f9411c8fa031c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39287 } health_report { overall_health: UNKNOWN } } }
I20260928 05:18:31.483628 11192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:31.478176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36736 (local address 127.0.97.1:37153)
0928 05:18:31.478614 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:31.478622 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:31.479134 (+ 512us) server_negotiation.cc:407] Connection header received
0928 05:18:31.479310 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:31.479314 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:31.479397 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:31.479531 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:18:31.479866 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:31.480520 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:31.481797 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:31.482174 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:31.482262 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:31.482663 (+ 401us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:18:31.483163 (+ 500us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:18:31.483260 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 05:18:31.483367 (+ 107us) server_negotiation.cc:299] Negotiation successful
0928 05:18:31.483434 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":200,"threads_started":1}
I20260928 05:18:31.490492 10982 raft_consensus.cc:1261] T 1fa9c6ee0cd9450fb11b479dd951546a P 949ca59be90f4906b1c2f632fbb3d9fb [term 1 FOLLOWER]: Refusing update from remote peer bf15c5852ece46dd9f2f6759ea6f254f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:18:31.490495 11115 raft_consensus.cc:1261] T 1fa9c6ee0cd9450fb11b479dd951546a P ecf559638fc047db83f9411c8fa031c6 [term 1 FOLLOWER]: Refusing update from remote peer bf15c5852ece46dd9f2f6759ea6f254f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:18:31.491283 11189 consensus_queue.cc:1035] T 1fa9c6ee0cd9450fb11b479dd951546a P bf15c5852ece46dd9f2f6759ea6f254f [LEADER]: Connected to new peer: Peer: permanent_uuid: "949ca59be90f4906b1c2f632fbb3d9fb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:18:31.491557 11184 consensus_queue.cc:1035] T 1fa9c6ee0cd9450fb11b479dd951546a P bf15c5852ece46dd9f2f6759ea6f254f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecf559638fc047db83f9411c8fa031c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:18:31.509066 11197 mvcc.cc:205] Tried to move back new op lower bound from 7334185826256621568 to 7334185825792061440. Current Snapshot: MvccSnapshot[applied={T|T < 7334185826256621568}]
I20260928 05:18:31.518800 10704 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41434:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:18:31.519039 10704 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41434:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:18:31.522104 10704 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c47a8cee0bfb4f858d7bf08b5b105c2a: Sending DeleteTablet for 3 replicas of tablet 1fa9c6ee0cd9450fb11b479dd951546a
I20260928 05:18:31.523066 10962 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fa9c6ee0cd9450fb11b479dd951546a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:18:31 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35172
I20260928 05:18:31.523095 10828 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fa9c6ee0cd9450fb11b479dd951546a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:18:31 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36724
I20260928 05:18:31.523422 11095 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fa9c6ee0cd9450fb11b479dd951546a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:18:31 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40286
I20260928 05:18:31.525069 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 10765
I20260928 05:18:31.533020 11206 tablet_replica.cc:330] T 1fa9c6ee0cd9450fb11b479dd951546a P 949ca59be90f4906b1c2f632fbb3d9fb: stopping tablet replica
I20260928 05:18:31.533942 11206 raft_consensus.cc:2229] T 1fa9c6ee0cd9450fb11b479dd951546a P 949ca59be90f4906b1c2f632fbb3d9fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:18:31.534931 11206 raft_consensus.cc:2258] T 1fa9c6ee0cd9450fb11b479dd951546a P 949ca59be90f4906b1c2f632fbb3d9fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:18:31.537335 11206 ts_tablet_manager.cc:1905] T 1fa9c6ee0cd9450fb11b479dd951546a P 949ca59be90f4906b1c2f632fbb3d9fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:18:31.545125 11207 tablet_replica.cc:330] T 1fa9c6ee0cd9450fb11b479dd951546a P ecf559638fc047db83f9411c8fa031c6: stopping tablet replica
I20260928 05:18:31.545401 11206 ts_tablet_manager.cc:1918] T 1fa9c6ee0cd9450fb11b479dd951546a P 949ca59be90f4906b1c2f632fbb3d9fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:18:31.545573 11206 log.cc:1196] T 1fa9c6ee0cd9450fb11b479dd951546a P 949ca59be90f4906b1c2f632fbb3d9fb: Deleting WAL directory at /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572622576160-388-0/minicluster-data/ts-1/wal/wals/1fa9c6ee0cd9450fb11b479dd951546a
I20260928 05:18:31.545984 11207 raft_consensus.cc:2229] T 1fa9c6ee0cd9450fb11b479dd951546a P ecf559638fc047db83f9411c8fa031c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:18:31.546113 11206 ts_tablet_manager.cc:1939] T 1fa9c6ee0cd9450fb11b479dd951546a P 949ca59be90f4906b1c2f632fbb3d9fb: Deleting consensus metadata
I20260928 05:18:31.547688 10688 catalog_manager.cc:4841] TS 949ca59be90f4906b1c2f632fbb3d9fb (127.0.97.2:42133): tablet 1fa9c6ee0cd9450fb11b479dd951546a (table test-table [id=8cbecdbd62e54d5f9c9aab696536a7c9]) successfully deleted
I20260928 05:18:31.548163 11207 raft_consensus.cc:2258] T 1fa9c6ee0cd9450fb11b479dd951546a P ecf559638fc047db83f9411c8fa031c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:18:31.549317 11205 tablet_replica.cc:330] T 1fa9c6ee0cd9450fb11b479dd951546a P bf15c5852ece46dd9f2f6759ea6f254f: stopping tablet replica
Sep 28 05:18:31 dist-test-slave-whsn krb5kdc[10659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572708, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260928 05:18:31.549732 11205 raft_consensus.cc:2229] T 1fa9c6ee0cd9450fb11b479dd951546a P bf15c5852ece46dd9f2f6759ea6f254f [term 1 LEADER]: Raft consensus shutting down.
I20260928 05:18:31.550266 11207 ts_tablet_manager.cc:1905] T 1fa9c6ee0cd9450fb11b479dd951546a P ecf559638fc047db83f9411c8fa031c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:18:31.550428 11205 raft_consensus.cc:2258] T 1fa9c6ee0cd9450fb11b479dd951546a P bf15c5852ece46dd9f2f6759ea6f254f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:18:31.552901 11205 ts_tablet_manager.cc:1905] T 1fa9c6ee0cd9450fb11b479dd951546a P bf15c5852ece46dd9f2f6759ea6f254f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:18:31.556806 11207 ts_tablet_manager.cc:1918] T 1fa9c6ee0cd9450fb11b479dd951546a P ecf559638fc047db83f9411c8fa031c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:18:31.557036 11207 log.cc:1196] T 1fa9c6ee0cd9450fb11b479dd951546a P ecf559638fc047db83f9411c8fa031c6: Deleting WAL directory at /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572622576160-388-0/minicluster-data/ts-2/wal/wals/1fa9c6ee0cd9450fb11b479dd951546a
I20260928 05:18:31.557436 11192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:31.526087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36748 (local address 127.0.97.1:37153)
0928 05:18:31.526247 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:31.526254 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:31.538371 (+ 12117us) server_negotiation.cc:407] Connection header received
0928 05:18:31.538764 (+ 393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:31.538769 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:31.538846 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:31.538910 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:18:31.540476 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:31.541612 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:31.543287 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:31.543931 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:31.544023 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:31.550613 (+ 6590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:18:31.550625 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:18:31.550640 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:18:31.550680 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:18:31.554178 (+ 3498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:18:31.554953 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:18:31.554960 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:18:31.554964 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:18:31.555130 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:18:31.555575 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:18:31.555580 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:18:31.555582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:18:31.555987 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:18:31.556240 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0928 05:18:31.556791 (+ 551us) server_negotiation.cc:299] Negotiation successful
0928 05:18:31.557062 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 05:18:31.557550 11207 ts_tablet_manager.cc:1939] T 1fa9c6ee0cd9450fb11b479dd951546a P ecf559638fc047db83f9411c8fa031c6: Deleting consensus metadata
I20260928 05:18:31.558980 10888 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:18:31.559988 10689 catalog_manager.cc:4841] TS ecf559638fc047db83f9411c8fa031c6 (127.0.97.3:39287): tablet 1fa9c6ee0cd9450fb11b479dd951546a (table test-table [id=8cbecdbd62e54d5f9c9aab696536a7c9]) successfully deleted
I20260928 05:18:31.651561 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 10765
I20260928 05:18:31.665452 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 10899
W20260928 05:18:31.665575 10690 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:37153 (error 108)
I20260928 05:18:31.667548 11211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:31.666744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37153 (local address 127.0.0.1:36752)
0928 05:18:31.667199 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0928 05:18:31.667319 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":237,"threads_started":1}
W20260928 05:18:31.667752 10690 catalog_manager.cc:4584] TS bf15c5852ece46dd9f2f6759ea6f254f (127.0.97.1:37153): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1fa9c6ee0cd9450fb11b479dd951546a: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37153: connect: Connection refused (error 111)
Sep 28 05:18:31 dist-test-slave-whsn krb5kdc[10659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572708, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260928 05:18:31.679157 11173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:31.666208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35180 (local address 127.0.97.2:42133)
0928 05:18:31.666369 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:31.666376 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:31.666433 (+ 57us) server_negotiation.cc:407] Connection header received
0928 05:18:31.666604 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:31.666609 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:31.666688 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:31.666769 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:18:31.668228 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:31.668981 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:31.670152 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:31.670533 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:31.670635 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:31.674192 (+ 3557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:18:31.674206 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:18:31.674219 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:18:31.674252 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:18:31.676808 (+ 2556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:18:31.677400 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:18:31.677406 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:18:31.677408 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:18:31.677532 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:18:31.677831 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:18:31.677836 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:18:31.677838 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:18:31.678163 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:18:31.678436 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0928 05:18:31.678673 (+ 237us) server_negotiation.cc:299] Negotiation successful
0928 05:18:31.678855 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 05:18:31.679728 11022 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:18:31.721100 11211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:31.720670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37153 (local address 127.0.0.1:36756)
0928 05:18:31.720842 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 05:18:31.720961 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260928 05:18:31.721980 11211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:31.721669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37153 (local address 127.0.0.1:36768)
0928 05:18:31.721783 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0928 05:18:31.721858 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260928 05:18:31.764374 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 10899
I20260928 05:18:31.778730 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 11032
I20260928 05:18:31.782239 11211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:31.781882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37153 (local address 127.0.0.1:36772)
0928 05:18:31.782042 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 05:18:31.782113 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260928 05:18:31.783106 11211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:31.782788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37153 (local address 127.0.0.1:36786)
0928 05:18:31.782901 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 05:18:31.782981 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
Sep 28 05:18:31 dist-test-slave-whsn krb5kdc[10659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572708, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260928 05:18:31.792181 11177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:31.779361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40302 (local address 127.0.97.3:39287)
0928 05:18:31.779528 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 05:18:31.779532 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:18:31.779550 (+ 18us) server_negotiation.cc:407] Connection header received
0928 05:18:31.779685 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:18:31.779689 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:18:31.779772 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:18:31.779848 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:18:31.780948 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:31.781693 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:31.783141 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:18:31.783563 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:18:31.783654 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:18:31.787425 (+ 3771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:18:31.787438 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:18:31.787453 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:18:31.787498 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:18:31.790072 (+ 2574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:18:31.790594 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:18:31.790600 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:18:31.790602 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:18:31.790723 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:18:31.790984 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:18:31.790987 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:18:31.790989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:18:31.791289 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:18:31.791501 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0928 05:18:31.791708 (+ 207us) server_negotiation.cc:299] Negotiation successful
0928 05:18:31.791852 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 05:18:31.792763 11155 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:18:31.863627 11211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:31.863129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37153 (local address 127.0.0.1:36802)
0928 05:18:31.863345 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0928 05:18:31.863449 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260928 05:18:31.864701 11211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:18:31.864265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37153 (local address 127.0.0.1:36816)
0928 05:18:31.864447 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 05:18:31.864525 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260928 05:18:31.884922 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 11032
I20260928 05:18:31.897591 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 10672
I20260928 05:18:31.898597 10734 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:18:32.001130 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 10672
2026-09-28T05:18:32Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4200 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:18:32 dist-test-slave-whsn krb5kdc[11217](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:18:32 dist-test-slave-whsn krb5kdc[11217](info): set up 2 sockets
Sep 28 05:18:32 dist-test-slave-whsn krb5kdc[11217](info): commencing operation
krb5kdc: starting...
W20260928 05:18:32.993505 388 mini_kdc.cc:121] Time spent starting KDC: real 0.956s user 0.005s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/krb5kdc/test-user.keytab.
Sep 28 05:18:33 dist-test-slave-whsn krb5kdc[11217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572713, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:18:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:18:33Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260928 05:18:33.257743 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-tasku1htJC/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/minicluster-data/postgres -l logfile start
2026-09-28 05:18:35.541 UTC [11247] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 05:18:35.541 UTC [11247] LOG: listening on IPv4 address "127.0.97.20", port 41929
2026-09-28 05:18:35.544 UTC [11247] LOG: listening on Unix socket "/tmp/.s.PGSQL.41929"
2026-09-28 05:18:35.564 UTC [11250] LOG: database system was shut down at 2026-09-28 05:18:34 UTC
2026-09-28 05:18:35.567 UTC [11247] LOG: database system is ready to accept connections
I20260928 05:18:36.461184 388 mini_postgres.cc:96] Postgres bound to 41929
2026-09-28 05:18:36.476 UTC [11258] FATAL: database "slave" does not exist
127.0.97.20:41929 - accepting connections
I20260928 05:18:36.477090 388 mini_ranger.cc:161] Starting Ranger...
I20260928 05:18:36.500765 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 05:18:36.912840 388 mini_ranger.cc:87] Created ranger Postgres database
I20260928 05:18:36.913012 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/minicluster-data/ranger-admin
2026-09-28 05:18:37,186 [I] DB FLAVOR :POSTGRES
2026-09-28 05:18:37,186 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 05:18:37,186 [I] Checking connection..
2026-09-28 05:18:37,186 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasku1htJC/build/asan/bin/postgresql.jar:/tmp/dist-test-tasku1htJC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41929/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 05:18:37,433 [I] Checking connection passed.
2026-09-28 05:18:37,433 [I] --------- Verifying version history table ---------
2026-09-28 05:18:37,433 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasku1htJC/build/asan/bin/postgresql.jar:/tmp/dist-test-tasku1htJC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41929/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 05:18:37,682 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 05:18:37,683 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasku1htJC/build/asan/bin/postgresql.jar:/tmp/dist-test-tasku1htJC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41929/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 05:18:37,927 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 05:18:37,927 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 05:18:37,928 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasku1htJC/build/asan/bin/postgresql.jar:/tmp/dist-test-tasku1htJC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41929/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasku1htJC/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 05:18:38.170 UTC [11322] WARNING: there is no transaction in progress
2026-09-28 05:18:38,175 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 05:18:38,176 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasku1htJC/build/asan/bin/postgresql.jar:/tmp/dist-test-tasku1htJC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41929/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 05:18:38,405 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 05:18:38,405 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 05:18:38,405 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasku1htJC/build/asan/bin/postgresql.jar:/tmp/dist-test-tasku1htJC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41929/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 05:18:38,639 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasku1htJC/build/asan/bin/postgresql.jar:/tmp/dist-test-tasku1htJC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41929/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 05:18:38,858 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasku1htJC/build/asan/bin/postgresql.jar:/tmp/dist-test-tasku1htJC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41929/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-whsn.c.gcp-upstream.internal','N') ;"
2026-09-28 05:18:39,094 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasku1htJC/build/asan/bin/postgresql.jar:/tmp/dist-test-tasku1htJC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41929/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 05:18:39,316 [I] Table x_portal_user does not exist in database ranger
2026-09-28 05:18:39,317 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasku1htJC/build/asan/bin/postgresql.jar:/tmp/dist-test-tasku1htJC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41929/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 05:18:39,534 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 05:18:39,535 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasku1htJC/build/asan/bin/postgresql.jar:/tmp/dist-test-tasku1htJC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41929/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 05:18:39,772 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasku1htJC/build/asan/bin/postgresql.jar:/tmp/dist-test-tasku1htJC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41929/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 05:18:40,019 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 05:18:40,019 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasku1htJC/build/asan/bin/postgresql.jar:/tmp/dist-test-tasku1htJC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41929/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasku1htJC/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 05:18:40.907 UTC [11430] WARNING: there is no transaction in progress
2026-09-28 05:18:40.921 UTC [11430] WARNING: there is no transaction in progress
2026-09-28 05:18:40.936 UTC [11430] WARNING: there is no transaction in progress
2026-09-28 05:18:40.949 UTC [11430] WARNING: there is no transaction in progress
2026-09-28 05:18:40.966 UTC [11430] WARNING: there is no transaction in progress
2026-09-28 05:18:40.985 UTC [11430] WARNING: there is no transaction in progress
2026-09-28 05:18:41.047 UTC [11430] WARNING: there is no transaction in progress
2026-09-28 05:18:41.053 UTC [11430] WARNING: there is no transaction in progress
2026-09-28 05:18:41.066 UTC [11430] WARNING: there is no transaction in progress
2026-09-28 05:18:41.075 UTC [11430] WARNING: there is no transaction in progress
2026-09-28 05:18:41.084 UTC [11430] WARNING: there is no transaction in progress
2026-09-28 05:18:41.097 UTC [11430] WARNING: there is no transaction in progress
2026-09-28 05:18:41.106 UTC [11430] WARNING: there is no transaction in progress
2026-09-28 05:18:41.112 UTC [11430] WARNING: there is no transaction in progress
2026-09-28 05:18:41.118 UTC [11430] WARNING: there is no transaction in progress
2026-09-28 05:18:41.124 UTC [11430] WARNING: there is no transaction in progress
2026-09-28 05:18:41,703 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 05:18:41,704 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasku1htJC/build/asan/bin/postgresql.jar:/tmp/dist-test-tasku1htJC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41929/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 05:18:41,930 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 05:18:41,930 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasku1htJC/build/asan/bin/postgresql.jar:/tmp/dist-test-tasku1htJC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41929/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 05:18:42,175 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 05:18:42,176 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasku1htJC/build/asan/bin/postgresql.jar:/tmp/dist-test-tasku1htJC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41929/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 05:18:42,420 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 05:18:42,420 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasku1htJC/build/asan/bin/postgresql.jar:/tmp/dist-test-tasku1htJC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41929/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 05:18:42,664 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasku1htJC/build/asan/bin/postgresql.jar:/tmp/dist-test-tasku1htJC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41929/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 05:18:42,907 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasku1htJC/build/asan/bin/postgresql.jar:/tmp/dist-test-tasku1htJC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41929/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-whsn.c.gcp-upstream.internal';"
2026-09-28 05:18:43,143 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 05:18:43,144 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasku1htJC/build/asan/bin/postgresql.jar:/tmp/dist-test-tasku1htJC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41929/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 05:18:43,393 [I] DB_PATCHES have already been applied
I20260928 05:18:43.398818 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/minicluster-data/ranger-admin:/tmp/dist-test-tasku1htJC/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-tasku1htJC/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-tasku1htJC/build/asan/bin/postgresql.jar:/tmp/dist-test-tasku1htJC/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 05:18:43.398924 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260928 05:18:43.412784 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:55393
Sep 28, 2026 5:18:43 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 5:18:43 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 5:18:43 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 5:18:43 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 5:18:43 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 5:18:44 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 5:18:44 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 05:18:44.436381 388 mini_ranger.cc:160] Time spent starting Ranger: real 7.959s user 0.009s sys 0.025s
Sep 28, 2026 5:19:03 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 5:19:03 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 28, 2026 5:19:03 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 5:19:03 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 5:19:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 5:19:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 5:19:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 5:19:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 5:19:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 5:19:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 5:19:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 5:19:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 5:19:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 5:19:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 5:19:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 5:19:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 5:19:03 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 5:19:04 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 05:19:05.242806 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 05:19:05 dist-test-slave-whsn krb5kdc[11217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572745, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 05:19:05.406749 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 05:19:05.424678 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 05:19:06.202244 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 05:19:06.202344 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/minicluster-data/ranger-kms
I20260928 05:19:06.202464 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:41929
2026-09-28 05:19:06,588 [I] DB FLAVOR :POSTGRES
2026-09-28 05:19:06,589 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 05:19:06,589 [I] Checking connection
2026-09-28 05:19:06,589 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasku1htJC/build/asan/bin/postgresql.jar:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41929/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 05:19:06,821 [I] connection success
2026-09-28 05:19:06,821 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 05:19:06,821 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 05:19:06,821 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasku1htJC/build/asan/bin/postgresql.jar:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41929/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 05:19:07,044 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 05:19:07,044 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 05:19:07,044 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 05:19:07,045 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasku1htJC/build/asan/bin/postgresql.jar:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41929/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 05:19:07,285 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 05:19:07.735066 388 mini_ranger_kms.cc:310] Created kms service
I20260928 05:19:07.908095 388 mini_ranger_kms.cc:326] Created kudu user
I20260928 05:19:07.950637 388 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 05:19:07 dist-test-slave-whsn krb5kdc[11217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572745, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260928 05:19:08.306785 388 mini_ranger_kms.cc:384] Added ranger policy
I20260928 05:19:08.307093 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/minicluster-data/ranger-kms:/tmp/dist-test-tasku1htJC/build/asan/bin/postgresql.jar:/tmp/dist-test-tasku1htJC/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-tasku1htJC/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 05:19:08.307152 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260928 05:19:08.325392 388 mini_ranger_kms.cc:276] Ranger KMS PID: 11610
I20260928 05:19:08.325591 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:59857
Sep 28, 2026 5:19:09 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 5:19:09 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 5:19:09 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 5:19:09 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 5:19:09 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 5:19:09 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 05:19:10.420665 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.014s user 0.119s sys 0.284s
I20260928 05:19:10.420923 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 05:19:10.421129 388 mini_ranger_kms.cc:401] 127.0.97.20:59857/kms/v1/keys
Sep 28 05:19:10 dist-test-slave-whsn krb5kdc[11217](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790572750, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 05:19:10 dist-test-slave-whsn krb5kdc[11217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572750, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 28 05:19:11 dist-test-slave-whsn krb5kdc[11217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572750, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 28 05:19:13 dist-test-slave-whsn krb5kdc[11217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572753, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 05:19:13.475745 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36475
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36475
--ranger_config_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:59857/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:19:13.811029 11661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:19:13.811292 11661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:19:13.811419 11661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:19:13.818285 11661 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:19:13.818384 11661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:19:13.818465 11661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:19:13.818535 11661 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:19:13.818593 11661 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:19:13.828626 11661 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:59857/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36475
--ranger_config_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:19:13.829833 11661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:19:13.831056 11661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:19:13.841658 11669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:19:13.841730 11666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:19:13.841706 11667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:19:13.842548 11661 server_base.cc:1034] running on GCE node
I20260928 05:19:13.843245 11661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:19:13.844514 11661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:19:13.846098 11661 hybrid_clock.cc:648] HybridClock initialized: now 1790572753845662 us; error 28 us; skew 500 ppm
I20260928 05:19:13.846415 11661 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:19:13 dist-test-slave-whsn krb5kdc[11217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572753, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:19:13.849205 11661 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 05:19:13.850463 11661 webserver.cc:460] Webserver started at http://127.0.97.62:35815/ using document root <none> and password file <none>
I20260928 05:19:13.851053 11661 fs_manager.cc:359] Metadata directory not provided
I20260928 05:19:13.851150 11661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:19:13.851411 11661 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 05:19:13 dist-test-slave-whsn krb5kdc[11217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572753, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260928 05:19:13.907269 11661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/minicluster-data/master-0/data/instance:
uuid: "32151fb09f4a40d3a227e8f053b7cf59"
format_stamp: "Formatted at 2026-09-28 05:19:13 on dist-test-slave-whsn"
server_key: "fdf0d03e43134c015c8bd2a0583426cc"
server_key_iv: "4baa0f1449eed73839ce5f16dc59fc88"
server_key_version: "kuduclusterkey@0"
I20260928 05:19:13.908116 11661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/minicluster-data/master-0/wal/instance:
uuid: "32151fb09f4a40d3a227e8f053b7cf59"
format_stamp: "Formatted at 2026-09-28 05:19:13 on dist-test-slave-whsn"
server_key: "fdf0d03e43134c015c8bd2a0583426cc"
server_key_iv: "4baa0f1449eed73839ce5f16dc59fc88"
server_key_version: "kuduclusterkey@0"
I20260928 05:19:13.914067 11661 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 05:19:13.932817 11676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:19:13.934176 11661 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 05:19:13.934326 11661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/minicluster-data/master-0/wal
uuid: "32151fb09f4a40d3a227e8f053b7cf59"
format_stamp: "Formatted at 2026-09-28 05:19:13 on dist-test-slave-whsn"
server_key: "fdf0d03e43134c015c8bd2a0583426cc"
server_key_iv: "4baa0f1449eed73839ce5f16dc59fc88"
server_key_version: "kuduclusterkey@0"
I20260928 05:19:13.934523 11661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:19:13.960562 11661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:19:13.964198 11661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:19:13.964462 11661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:19:13.983633 11661 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36475
I20260928 05:19:13.983654 11727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36475 every 8 connection(s)
I20260928 05:19:13.984987 11661 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:19:13.989058 11728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:19:13.991041 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 11661
I20260928 05:19:13.991233 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/minicluster-data/master-0/wal/instance
I20260928 05:19:13.991946 388 external_mini_cluster.cc:1428] Setting key d7dafa146939662b76a1f88a721e0ce6
I20260928 05:19:14.000389 11728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32151fb09f4a40d3a227e8f053b7cf59: Bootstrap starting.
I20260928 05:19:14.004417 11728 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 32151fb09f4a40d3a227e8f053b7cf59: Neither blocks nor log segments found. Creating new log.
Sep 28 05:19:14 dist-test-slave-whsn krb5kdc[11217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572753, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:19:14.005465 11728 log.cc:824] T 00000000000000000000000000000000 P 32151fb09f4a40d3a227e8f053b7cf59: Log is configured to *not* fsync() on all Append() calls
I20260928 05:19:14.008203 11728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32151fb09f4a40d3a227e8f053b7cf59: No bootstrap required, opened a new log
I20260928 05:19:14.012758 11731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:19:13.993899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58086 (local address 127.0.97.62:36475)
0928 05:19:13.994624 (+ 725us) server_negotiation.cc:206] Beginning negotiation
0928 05:19:13.994635 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 05:19:13.994675 (+ 40us) server_negotiation.cc:407] Connection header received
0928 05:19:13.996221 (+ 1546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:19:13.996242 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:19:13.996615 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:19:13.996967 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:19:13.998332 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:19:13.999592 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:19:14.001230 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:19:14.001865 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:19:14.002010 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:19:14.006150 (+ 4140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:19:14.006157 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:19:14.006174 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:19:14.006215 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:19:14.009135 (+ 2920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:19:14.009742 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:19:14.009748 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:19:14.009757 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:19:14.009905 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:19:14.010262 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:19:14.010265 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:19:14.010267 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:19:14.010772 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:19:14.011052 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0928 05:19:14.011465 (+ 413us) server_negotiation.cc:299] Negotiation successful
0928 05:19:14.011819 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":319,"threads_started":1}
I20260928 05:19:14.015905 11728 raft_consensus.cc:348] T 00000000000000000000000000000000 P 32151fb09f4a40d3a227e8f053b7cf59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32151fb09f4a40d3a227e8f053b7cf59" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36475 } }
I20260928 05:19:14.016232 11728 raft_consensus.cc:374] T 00000000000000000000000000000000 P 32151fb09f4a40d3a227e8f053b7cf59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:19:14.016316 11728 raft_consensus.cc:729] T 00000000000000000000000000000000 P 32151fb09f4a40d3a227e8f053b7cf59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32151fb09f4a40d3a227e8f053b7cf59, State: Initialized, Role: FOLLOWER
I20260928 05:19:14.016826 11728 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32151fb09f4a40d3a227e8f053b7cf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32151fb09f4a40d3a227e8f053b7cf59" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36475 } }
I20260928 05:19:14.017089 11728 raft_consensus.cc:388] T 00000000000000000000000000000000 P 32151fb09f4a40d3a227e8f053b7cf59 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:19:14.017205 11728 raft_consensus.cc:482] T 00000000000000000000000000000000 P 32151fb09f4a40d3a227e8f053b7cf59 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:19:14.017346 11728 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 32151fb09f4a40d3a227e8f053b7cf59 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:19:14.019218 11728 raft_consensus.cc:504] T 00000000000000000000000000000000 P 32151fb09f4a40d3a227e8f053b7cf59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32151fb09f4a40d3a227e8f053b7cf59" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36475 } }
I20260928 05:19:14.019661 11728 leader_election.cc:302] T 00000000000000000000000000000000 P 32151fb09f4a40d3a227e8f053b7cf59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 32151fb09f4a40d3a227e8f053b7cf59; no voters:
I20260928 05:19:14.020136 11728 leader_election.cc:288] T 00000000000000000000000000000000 P 32151fb09f4a40d3a227e8f053b7cf59 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:19:14.020236 11733 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 32151fb09f4a40d3a227e8f053b7cf59 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:19:14.020804 11733 raft_consensus.cc:686] T 00000000000000000000000000000000 P 32151fb09f4a40d3a227e8f053b7cf59 [term 1 LEADER]: Becoming Leader. State: Replica: 32151fb09f4a40d3a227e8f053b7cf59, State: Running, Role: LEADER
I20260928 05:19:14.021540 11733 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32151fb09f4a40d3a227e8f053b7cf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32151fb09f4a40d3a227e8f053b7cf59" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36475 } }
I20260928 05:19:14.022083 11728 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32151fb09f4a40d3a227e8f053b7cf59 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:19:14.024823 11735 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32151fb09f4a40d3a227e8f053b7cf59 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "32151fb09f4a40d3a227e8f053b7cf59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32151fb09f4a40d3a227e8f053b7cf59" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36475 } } }
I20260928 05:19:14.024955 11734 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32151fb09f4a40d3a227e8f053b7cf59 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32151fb09f4a40d3a227e8f053b7cf59. Latest consensus state: current_term: 1 leader_uuid: "32151fb09f4a40d3a227e8f053b7cf59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32151fb09f4a40d3a227e8f053b7cf59" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36475 } } }
I20260928 05:19:14.025249 11734 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32151fb09f4a40d3a227e8f053b7cf59 [sys.catalog]: This master's current role is: LEADER
I20260928 05:19:14.025247 11735 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32151fb09f4a40d3a227e8f053b7cf59 [sys.catalog]: This master's current role is: LEADER
I20260928 05:19:14.029436 11740 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:19:14.034593 11740 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:19:14.035087 11728 ranger_client.cc:318] Using new properties file: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 05:19:14.051318 11740 catalog_manager.cc:1347] Generated new cluster ID: 208ed5ae4ac34337a98f4c8e27d97139
I20260928 05:19:14.051443 11740 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:19:14.082885 11740 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:19:14.084931 11740 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:19:14.093044 11740 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 32151fb09f4a40d3a227e8f053b7cf59: Generated new TSK 0
I20260928 05:19:14.094046 11740 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260928 05:19:14.779485 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:59857/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36475
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 05:19:15 dist-test-slave-whsn krb5kdc[11217](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790572755, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260928 05:19:15.168643 11773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:19:15.169039 11773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:19:15.169198 11773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:19:15.180356 11773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:19:15.180533 11773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:19:15.180742 11773 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 05:19:15.195979 11773 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:59857/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36475
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:19:15.197587 11773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:19:15.199234 11773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:19:15.214679 11783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:19:15.216053 11784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:19:15.223590 11786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:19:15.224061 11773 server_base.cc:1034] running on GCE node
I20260928 05:19:15.224730 11773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:19:15.225900 11773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:19:15.227211 11773 hybrid_clock.cc:648] HybridClock initialized: now 1790572755227111 us; error 88 us; skew 500 ppm
I20260928 05:19:15.227612 11773 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:19:15 dist-test-slave-whsn krb5kdc[11217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572755, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:19:15.233894 11773 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260928 05:19:15.240121 11773 webserver.cc:460] Webserver started at http://127.0.97.1:38625/ using document root <none> and password file <none>
I20260928 05:19:15.241117 11773 fs_manager.cc:359] Metadata directory not provided
I20260928 05:19:15.241304 11773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:19:15.241701 11773 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 05:19:15 dist-test-slave-whsn krb5kdc[11217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572755, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260928 05:19:15.258319 11773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/minicluster-data/ts-0/data/instance:
uuid: "ca52468b263a4d3283aca23c0771c185"
format_stamp: "Formatted at 2026-09-28 05:19:15 on dist-test-slave-whsn"
server_key: "51805b2394c3464482774f4ba2e7a2b2"
server_key_iv: "ee382fa8042b81f4eee9f68ae395ecd6"
server_key_version: "kuduclusterkey@0"
I20260928 05:19:15.259099 11773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/minicluster-data/ts-0/wal/instance:
uuid: "ca52468b263a4d3283aca23c0771c185"
format_stamp: "Formatted at 2026-09-28 05:19:15 on dist-test-slave-whsn"
server_key: "51805b2394c3464482774f4ba2e7a2b2"
server_key_iv: "ee382fa8042b81f4eee9f68ae395ecd6"
server_key_version: "kuduclusterkey@0"
I20260928 05:19:15.274268 11773 fs_manager.cc:691] Time spent creating directory manager: real 0.015s user 0.004s sys 0.012s
I20260928 05:19:15.290109 11793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:19:15.291589 11773 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 05:19:15.291739 11773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/minicluster-data/ts-0/wal
uuid: "ca52468b263a4d3283aca23c0771c185"
format_stamp: "Formatted at 2026-09-28 05:19:15 on dist-test-slave-whsn"
server_key: "51805b2394c3464482774f4ba2e7a2b2"
server_key_iv: "ee382fa8042b81f4eee9f68ae395ecd6"
server_key_version: "kuduclusterkey@0"
I20260928 05:19:15.291940 11773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:19:15.324977 11773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:19:15.329912 11773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:19:15.330274 11773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:19:15.334187 11773 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:19:15.343312 11773 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:19:15.343391 11773 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:19:15.343544 11773 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:19:15.343657 11773 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 05:19:15 dist-test-slave-whsn krb5kdc[11217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572755, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260928 05:19:15.719137 11773 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40649
I20260928 05:19:15.719167 11905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40649 every 8 connection(s)
I20260928 05:19:15.720767 11773 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/minicluster-data/ts-0/data/info.pb
I20260928 05:19:15.727906 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 11773
I20260928 05:19:15.728104 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/minicluster-data/ts-0/wal/instance
I20260928 05:19:15.728878 388 external_mini_cluster.cc:1428] Setting key 7baa7109bee96c6ea85d656188cd8898
Sep 28 05:19:15 dist-test-slave-whsn krb5kdc[11217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572755, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:19:15.748696 11909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:19:15.724247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36475 (local address 127.0.97.1:59625)
0928 05:19:15.725052 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0928 05:19:15.725097 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 05:19:15.726706 (+ 1609us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:19:15.727402 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:19:15.727411 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:19:15.727842 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:19:15.728470 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:19:15.728491 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:19:15.729616 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:19:15.729622 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:19:15.732049 (+ 2427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:19:15.732074 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:19:15.733041 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:19:15.733048 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:19:15.733317 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:19:15.734020 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:19:15.734048 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:19:15.737133 (+ 3085us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:19:15.740301 (+ 3168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:19:15.740310 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:19:15.740325 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:19:15.740719 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:19:15.745002 (+ 4283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:19:15.745008 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:19:15.745012 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:19:15.745302 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:19:15.746059 (+ 757us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:19:15.746068 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:19:15.746505 (+ 437us) client_negotiation.cc:769] Sending connection context
0928 05:19:15.746818 (+ 313us) client_negotiation.cc:240] Negotiation successful
0928 05:19:15.747323 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":309,"threads_started":1}
I20260928 05:19:15.749744 11908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:19:15.723466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59625 (local address 127.0.97.62:36475)
0928 05:19:15.723916 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0928 05:19:15.723920 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:19:15.725201 (+ 1281us) server_negotiation.cc:407] Connection header received
0928 05:19:15.726973 (+ 1772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:19:15.726978 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:19:15.727069 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:19:15.727165 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:19:15.728636 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:19:15.729420 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:19:15.732219 (+ 2799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:19:15.732784 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:19:15.732923 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:19:15.737368 (+ 4445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:19:15.737374 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:19:15.737379 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:19:15.737426 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:19:15.740173 (+ 2747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:19:15.740846 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:19:15.740875 (+ 29us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:19:15.740879 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:19:15.741012 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:19:15.745490 (+ 4478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:19:15.745496 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:19:15.745499 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:19:15.745845 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:19:15.745973 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 05:19:15.749209 (+ 3236us) server_negotiation.cc:299] Negotiation successful
0928 05:19:15.749425 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":220,"threads_started":1}
I20260928 05:19:15.751080 11906 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36475
I20260928 05:19:15.751384 11906 heartbeater.cc:464] Registering TS with master...
I20260928 05:19:15.752205 11906 heartbeater.cc:511] Master 127.0.97.62:36475 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260928 05:19:15.754652 11693 ts_manager.cc:194] Registered new tserver with Master: ca52468b263a4d3283aca23c0771c185 (127.0.97.1:40649)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260928 05:19:15.756639 11693 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:59625
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260928 05:19:15.785553 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:59857/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36475
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 05:19:15 dist-test-slave-whsn krb5kdc[11217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572755, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260928 05:19:16.217871 11912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:19:16.218134 11912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:19:16.218262 11912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:19:16.225723 11912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:19:16.225834 11912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:19:16.225973 11912 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 05:19:16.235414 11912 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:59857/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36475
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:19:16.236591 11912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:19:16.237919 11912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:19:16.252214 11926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:19:16.252312 11927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:19:16.252686 11929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:19:16.253175 11912 server_base.cc:1034] running on GCE node
I20260928 05:19:16.253710 11912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:19:16.254611 11912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:19:16.255815 11912 hybrid_clock.cc:648] HybridClock initialized: now 1790572756255757 us; error 50 us; skew 500 ppm
I20260928 05:19:16.256110 11912 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:19:16 dist-test-slave-whsn krb5kdc[11217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:19:16.259290 11912 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260928 05:19:16.260635 11912 webserver.cc:460] Webserver started at http://127.0.97.2:36095/ using document root <none> and password file <none>
I20260928 05:19:16.261265 11912 fs_manager.cc:359] Metadata directory not provided
I20260928 05:19:16.261363 11912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:19:16.261644 11912 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 05:19:16 dist-test-slave-whsn krb5kdc[11217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260928 05:19:16.278371 11912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/minicluster-data/ts-1/data/instance:
uuid: "fd46289041554333896772e7fc1e0efd"
format_stamp: "Formatted at 2026-09-28 05:19:16 on dist-test-slave-whsn"
server_key: "83a48852492ec917a0dcb32995ad2365"
server_key_iv: "64e6f7bf3710b715b7210fea01bb5bb3"
server_key_version: "kuduclusterkey@0"
I20260928 05:19:16.279485 11912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/minicluster-data/ts-1/wal/instance:
uuid: "fd46289041554333896772e7fc1e0efd"
format_stamp: "Formatted at 2026-09-28 05:19:16 on dist-test-slave-whsn"
server_key: "83a48852492ec917a0dcb32995ad2365"
server_key_iv: "64e6f7bf3710b715b7210fea01bb5bb3"
server_key_version: "kuduclusterkey@0"
I20260928 05:19:16.286530 11912 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.004s
I20260928 05:19:16.304685 11936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:19:16.306628 11912 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 05:19:16.306797 11912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/minicluster-data/ts-1/wal
uuid: "fd46289041554333896772e7fc1e0efd"
format_stamp: "Formatted at 2026-09-28 05:19:16 on dist-test-slave-whsn"
server_key: "83a48852492ec917a0dcb32995ad2365"
server_key_iv: "64e6f7bf3710b715b7210fea01bb5bb3"
server_key_version: "kuduclusterkey@0"
I20260928 05:19:16.307030 11912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:19:16.330838 11912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:19:16.335445 11912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:19:16.335747 11912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:19:16.336830 11912 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:19:16.338599 11912 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:19:16.338682 11912 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:19:16.338842 11912 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:19:16.338902 11912 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:19:16.383304 11912 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40053
I20260928 05:19:16.383328 12048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40053 every 8 connection(s)
I20260928 05:19:16.384938 11912 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/minicluster-data/ts-1/data/info.pb
I20260928 05:19:16.386785 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 11912
I20260928 05:19:16.386952 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/minicluster-data/ts-1/wal/instance
I20260928 05:19:16.387434 388 external_mini_cluster.cc:1428] Setting key a98ea2786304e33d8af69903bf87094f
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Sep 28 05:19:16 dist-test-slave-whsn krb5kdc[11217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:19:16.432646 12051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:19:16.389720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49861 (local address 127.0.97.62:36475)
0928 05:19:16.401233 (+ 11513us) server_negotiation.cc:206] Beginning negotiation
0928 05:19:16.401239 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:19:16.409380 (+ 8141us) server_negotiation.cc:407] Connection header received
0928 05:19:16.412930 (+ 3550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:19:16.412936 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:19:16.413032 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:19:16.413134 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:19:16.414947 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:19:16.416074 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:19:16.418439 (+ 2365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:19:16.419071 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:19:16.419191 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:19:16.425398 (+ 6207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:19:16.425405 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:19:16.425410 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:19:16.425462 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:19:16.428934 (+ 3472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:19:16.429897 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:19:16.429903 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:19:16.429907 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:19:16.430046 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:19:16.430545 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:19:16.430550 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:19:16.430554 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:19:16.430906 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:19:16.431029 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 05:19:16.432024 (+ 995us) server_negotiation.cc:299] Negotiation successful
0928 05:19:16.432252 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11371,"thread_start_us":11252,"threads_started":1}
I20260928 05:19:16.435043 12052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:19:16.390369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36475 (local address 127.0.97.2:49861)
0928 05:19:16.409210 (+ 18841us) negotiation.cc:107] Waiting for socket to connect
0928 05:19:16.409267 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 05:19:16.412590 (+ 3323us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:19:16.413394 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:19:16.413407 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:19:16.413980 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:19:16.414755 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:19:16.414780 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:19:16.416242 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:19:16.416247 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:19:16.418264 (+ 2017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:19:16.418286 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:19:16.419485 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:19:16.419493 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:19:16.419785 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:19:16.420525 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:19:16.420554 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:19:16.425111 (+ 4557us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:19:16.429131 (+ 4020us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:19:16.429140 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:19:16.429159 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:19:16.429761 (+ 602us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:19:16.430181 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:19:16.430186 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:19:16.430190 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:19:16.430416 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:19:16.431115 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:19:16.431123 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:19:16.431541 (+ 418us) client_negotiation.cc:769] Sending connection context
0928 05:19:16.433198 (+ 1657us) client_negotiation.cc:240] Negotiation successful
0928 05:19:16.433722 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18541,"thread_start_us":20560,"threads_started":1}
I20260928 05:19:16.436941 12049 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36475
I20260928 05:19:16.437233 12049 heartbeater.cc:464] Registering TS with master...
I20260928 05:19:16.438048 12049 heartbeater.cc:511] Master 127.0.97.62:36475 requested a full tablet report, sending...
I20260928 05:19:16.439625 11693 ts_manager.cc:194] Registered new tserver with Master: fd46289041554333896772e7fc1e0efd (127.0.97.2:40053)
I20260928 05:19:16.440696 11693 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:49861
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260928 05:19:16.454064 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:59857/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36475
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 05:19:16.760774 11906 heartbeater.cc:503] Master 127.0.97.62:36475 was elected leader, sending a full tablet report...
W20260928 05:19:16.812996 12055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:19:16.813235 12055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:19:16.813321 12055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:19:16.820111 12055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:19:16.820199 12055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:19:16.820302 12055 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 05:19:16.829824 12055 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:59857/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36475
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:19:16.830979 12055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:19:16.832119 12055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:19:16.844545 12063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:19:16.844590 12061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:19:16.844612 12060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:19:16.845253 12055 server_base.cc:1034] running on GCE node
I20260928 05:19:16.845722 12055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:19:16.846496 12055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:19:16.847671 12055 hybrid_clock.cc:648] HybridClock initialized: now 1790572756847633 us; error 32 us; skew 500 ppm
I20260928 05:19:16.847939 12055 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:19:16 dist-test-slave-whsn krb5kdc[11217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:19:16.850679 12055 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260928 05:19:16.851924 12055 webserver.cc:460] Webserver started at http://127.0.97.3:45311/ using document root <none> and password file <none>
I20260928 05:19:16.852501 12055 fs_manager.cc:359] Metadata directory not provided
I20260928 05:19:16.852607 12055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:19:16.852905 12055 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 05:19:16 dist-test-slave-whsn krb5kdc[11217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260928 05:19:16.868537 12055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/minicluster-data/ts-2/data/instance:
uuid: "d8662becc5334b65bbef6990a7ac4a44"
format_stamp: "Formatted at 2026-09-28 05:19:16 on dist-test-slave-whsn"
server_key: "34ecd8b01228cb85bacd0e27c1be1401"
server_key_iv: "4b6fd39d4fc4b1617c3d9e6659843722"
server_key_version: "kuduclusterkey@0"
I20260928 05:19:16.869591 12055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/minicluster-data/ts-2/wal/instance:
uuid: "d8662becc5334b65bbef6990a7ac4a44"
format_stamp: "Formatted at 2026-09-28 05:19:16 on dist-test-slave-whsn"
server_key: "34ecd8b01228cb85bacd0e27c1be1401"
server_key_iv: "4b6fd39d4fc4b1617c3d9e6659843722"
server_key_version: "kuduclusterkey@0"
I20260928 05:19:16.875802 12055 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260928 05:19:16.892107 12070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:19:16.893808 12055 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:19:16.893985 12055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/minicluster-data/ts-2/wal
uuid: "d8662becc5334b65bbef6990a7ac4a44"
format_stamp: "Formatted at 2026-09-28 05:19:16 on dist-test-slave-whsn"
server_key: "34ecd8b01228cb85bacd0e27c1be1401"
server_key_iv: "4b6fd39d4fc4b1617c3d9e6659843722"
server_key_version: "kuduclusterkey@0"
I20260928 05:19:16.894235 12055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:19:16.932741 12055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:19:16.937134 12055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:19:16.937427 12055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:19:16.938588 12055 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:19:16.940222 12055 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:19:16.940306 12055 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:19:16.940418 12055 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:19:16.940495 12055 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:19:16.971290 12055 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45479
I20260928 05:19:16.971318 12182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45479 every 8 connection(s)
I20260928 05:19:16.972795 12055 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/minicluster-data/ts-2/data/info.pb
I20260928 05:19:16.981772 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 12055
I20260928 05:19:16.981936 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/minicluster-data/ts-2/wal/instance
I20260928 05:19:16.982432 388 external_mini_cluster.cc:1428] Setting key 1ec6f29a3802e1af90e7240deb943e2b
Sep 28 05:19:16 dist-test-slave-whsn krb5kdc[11217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:19:16.992475 12185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:19:16.975283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55637 (local address 127.0.97.62:36475)
0928 05:19:16.975663 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0928 05:19:16.975668 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:19:16.977348 (+ 1680us) server_negotiation.cc:407] Connection header received
0928 05:19:16.979038 (+ 1690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:19:16.979044 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:19:16.979127 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:19:16.979227 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:19:16.980550 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:19:16.981289 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:19:16.982870 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:19:16.983644 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:19:16.983767 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:19:16.987248 (+ 3481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:19:16.987252 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:19:16.987256 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:19:16.987298 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:19:16.989768 (+ 2470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:19:16.990378 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:19:16.990382 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:19:16.990384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:19:16.990507 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:19:16.990856 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:19:16.990861 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:19:16.990864 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:19:16.991098 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:19:16.991216 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 05:19:16.991909 (+ 693us) server_negotiation.cc:299] Negotiation successful
0928 05:19:16.992131 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":191,"threads_started":1}
I20260928 05:19:16.993382 12186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:19:16.976415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36475 (local address 127.0.97.3:55637)
0928 05:19:16.977198 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0928 05:19:16.977238 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 05:19:16.978778 (+ 1540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:19:16.979454 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:19:16.979462 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:19:16.979863 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:19:16.980399 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:19:16.980415 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:19:16.981433 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:19:16.981438 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:19:16.982734 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:19:16.982752 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:19:16.984048 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:19:16.984055 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:19:16.984335 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:19:16.985094 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:19:16.985123 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:19:16.987066 (+ 1943us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:19:16.989882 (+ 2816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:19:16.989891 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:19:16.989906 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:19:16.990271 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:19:16.990601 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:19:16.990606 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:19:16.990609 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:19:16.990760 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:19:16.991214 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:19:16.991221 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:19:16.991477 (+ 256us) client_negotiation.cc:769] Sending connection context
0928 05:19:16.991807 (+ 330us) client_negotiation.cc:240] Negotiation successful
0928 05:19:16.992244 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":338,"threads_started":1}
I20260928 05:19:16.994580 12183 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36475
I20260928 05:19:16.994838 12183 heartbeater.cc:464] Registering TS with master...
I20260928 05:19:16.995527 12183 heartbeater.cc:511] Master 127.0.97.62:36475 requested a full tablet report, sending...
I20260928 05:19:16.996975 11693 ts_manager.cc:194] Registered new tserver with Master: d8662becc5334b65bbef6990a7ac4a44 (127.0.97.3:45479)
I20260928 05:19:16.997659 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 05:19:16.997915 11693 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:55637
I20260928 05:19:17.017558 12185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:19:17.002978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58092 (local address 127.0.97.62:36475)
0928 05:19:17.003152 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 05:19:17.003159 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:19:17.005419 (+ 2260us) server_negotiation.cc:407] Connection header received
0928 05:19:17.005592 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:19:17.005598 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:19:17.005660 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:19:17.005732 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:19:17.007390 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:19:17.008038 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:19:17.009920 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:19:17.010313 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:19:17.010397 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:19:17.012979 (+ 2582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:19:17.012984 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:19:17.012987 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:19:17.013017 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:19:17.015309 (+ 2292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:19:17.015972 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:19:17.015975 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:19:17.015977 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:19:17.016074 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:19:17.016408 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:19:17.016410 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:19:17.016412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:19:17.016655 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:19:17.016810 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 05:19:17.017170 (+ 360us) server_negotiation.cc:299] Negotiation successful
0928 05:19:17.017315 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 05:19:17.021742 11693 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58092:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 05:19:17.028502 11693 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 05:19:17.050257 12196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:19:17.040802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40649 (local address 127.0.0.1:39978)
0928 05:19:17.041453 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0928 05:19:17.041494 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 05:19:17.041648 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:19:17.042408 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:19:17.042415 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:19:17.042444 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:19:17.042833 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:19:17.042847 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:19:17.045038 (+ 2191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:19:17.045044 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:19:17.047240 (+ 2196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:19:17.047256 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:19:17.049330 (+ 2074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:19:17.049336 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:19:17.049539 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:19:17.049587 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 05:19:17.049705 (+ 118us) client_negotiation.cc:240] Negotiation successful
0928 05:19:17.049882 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":623,"threads_started":1}
I20260928 05:19:17.051487 12197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:19:17.041244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39978 (local address 127.0.97.1:40649)
0928 05:19:17.041827 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0928 05:19:17.041836 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:19:17.041863 (+ 27us) server_negotiation.cc:407] Connection header received
0928 05:19:17.041959 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:19:17.041966 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:19:17.042207 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:19:17.042382 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:19:17.043035 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:19:17.044887 (+ 1852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:19:17.047430 (+ 2543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:19:17.048845 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:19:17.048985 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:19:17.049207 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0928 05:19:17.051002 (+ 1795us) server_negotiation.cc:299] Negotiation successful
0928 05:19:17.051173 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":675,"threads_started":1}
I20260928 05:19:17.056947 11841 tablet_service.cc:1467] Processing CreateTablet for tablet a59c5115d4da4d179314104f1c793bab (DEFAULT_TABLE table=test-table [id=647e3abe038b49ef8b4636dcbf35906e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:19:17.058487 11841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a59c5115d4da4d179314104f1c793bab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:19:17.059687 12196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:19:17.042670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40053 (local address 127.0.0.1:41228)
0928 05:19:17.050710 (+ 8040us) negotiation.cc:107] Waiting for socket to connect
0928 05:19:17.050742 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 05:19:17.050847 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:19:17.052877 (+ 2030us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:19:17.052882 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:19:17.052907 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:19:17.053075 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:19:17.053086 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:19:17.055152 (+ 2066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:19:17.055158 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:19:17.057129 (+ 1971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:19:17.057144 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:19:17.058951 (+ 1807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:19:17.058957 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:19:17.059166 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:19:17.059208 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 05:19:17.059269 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 05:19:17.059402 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7934,"threads_started":1}
I20260928 05:19:17.060761 12199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:19:17.042893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41228 (local address 127.0.97.2:40053)
0928 05:19:17.052272 (+ 9379us) server_negotiation.cc:206] Beginning negotiation
0928 05:19:17.052281 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:19:17.052309 (+ 28us) server_negotiation.cc:407] Connection header received
0928 05:19:17.052399 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:19:17.052407 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:19:17.052660 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:19:17.053249 (+ 589us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:19:17.053449 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:19:17.054980 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:19:17.057331 (+ 2351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:19:17.058772 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:19:17.060036 (+ 1264us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:19:17.060243 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0928 05:19:17.060326 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 05:19:17.060482 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9227,"thread_start_us":9102,"threads_started":1}
I20260928 05:19:17.061898 12201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:19:17.044947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45654 (local address 127.0.97.3:45479)
0928 05:19:17.052718 (+ 7771us) server_negotiation.cc:206] Beginning negotiation
0928 05:19:17.052726 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:19:17.053386 (+ 660us) server_negotiation.cc:407] Connection header received
0928 05:19:17.053470 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:19:17.053477 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:19:17.053691 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:19:17.053850 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:19:17.055205 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:19:17.056590 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:19:17.058840 (+ 2250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:19:17.060107 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:19:17.060219 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:19:17.060411 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0928 05:19:17.061439 (+ 1028us) server_negotiation.cc:299] Negotiation successful
0928 05:19:17.061592 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7579,"thread_start_us":7423,"threads_started":1}
I20260928 05:19:17.062233 12200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:19:17.044411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45479 (local address 127.0.0.1:45654)
0928 05:19:17.053160 (+ 8749us) negotiation.cc:107] Waiting for socket to connect
0928 05:19:17.053177 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 05:19:17.053285 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:19:17.054084 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:19:17.054088 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:19:17.054112 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:19:17.054986 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:19:17.055000 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:19:17.056736 (+ 1736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:19:17.056740 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:19:17.058678 (+ 1938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:19:17.058690 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:19:17.060797 (+ 2107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:19:17.060803 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:19:17.060979 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:19:17.061010 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 05:19:17.061091 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 05:19:17.061198 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8632,"thread_start_us":8484,"threads_started":1}
I20260928 05:19:17.064450 11984 tablet_service.cc:1467] Processing CreateTablet for tablet a59c5115d4da4d179314104f1c793bab (DEFAULT_TABLE table=test-table [id=647e3abe038b49ef8b4636dcbf35906e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:19:17.065794 11984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a59c5115d4da4d179314104f1c793bab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:19:17.065549 12118 tablet_service.cc:1467] Processing CreateTablet for tablet a59c5115d4da4d179314104f1c793bab (DEFAULT_TABLE table=test-table [id=647e3abe038b49ef8b4636dcbf35906e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:19:17.066813 12118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a59c5115d4da4d179314104f1c793bab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:19:17.082615 12202 tablet_bootstrap.cc:492] T a59c5115d4da4d179314104f1c793bab P ca52468b263a4d3283aca23c0771c185: Bootstrap starting.
I20260928 05:19:17.084324 12204 tablet_bootstrap.cc:492] T a59c5115d4da4d179314104f1c793bab P d8662becc5334b65bbef6990a7ac4a44: Bootstrap starting.
I20260928 05:19:17.086892 12202 tablet_bootstrap.cc:654] T a59c5115d4da4d179314104f1c793bab P ca52468b263a4d3283aca23c0771c185: Neither blocks nor log segments found. Creating new log.
I20260928 05:19:17.087845 12204 tablet_bootstrap.cc:654] T a59c5115d4da4d179314104f1c793bab P d8662becc5334b65bbef6990a7ac4a44: Neither blocks nor log segments found. Creating new log.
I20260928 05:19:17.088586 12202 log.cc:824] T a59c5115d4da4d179314104f1c793bab P ca52468b263a4d3283aca23c0771c185: Log is configured to *not* fsync() on all Append() calls
I20260928 05:19:17.091439 12203 tablet_bootstrap.cc:492] T a59c5115d4da4d179314104f1c793bab P fd46289041554333896772e7fc1e0efd: Bootstrap starting.
I20260928 05:19:17.091666 12202 tablet_bootstrap.cc:492] T a59c5115d4da4d179314104f1c793bab P ca52468b263a4d3283aca23c0771c185: No bootstrap required, opened a new log
I20260928 05:19:17.092023 12202 ts_tablet_manager.cc:1397] T a59c5115d4da4d179314104f1c793bab P ca52468b263a4d3283aca23c0771c185: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260928 05:19:17.095120 12204 log.cc:824] T a59c5115d4da4d179314104f1c793bab P d8662becc5334b65bbef6990a7ac4a44: Log is configured to *not* fsync() on all Append() calls
I20260928 05:19:17.095643 12203 tablet_bootstrap.cc:654] T a59c5115d4da4d179314104f1c793bab P fd46289041554333896772e7fc1e0efd: Neither blocks nor log segments found. Creating new log.
I20260928 05:19:17.096988 12203 log.cc:824] T a59c5115d4da4d179314104f1c793bab P fd46289041554333896772e7fc1e0efd: Log is configured to *not* fsync() on all Append() calls
I20260928 05:19:17.103703 12202 raft_consensus.cc:348] T a59c5115d4da4d179314104f1c793bab P ca52468b263a4d3283aca23c0771c185 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca52468b263a4d3283aca23c0771c185" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40649 } } peers { permanent_uuid: "fd46289041554333896772e7fc1e0efd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40053 } } peers { permanent_uuid: "d8662becc5334b65bbef6990a7ac4a44" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45479 } }
I20260928 05:19:17.104276 12202 raft_consensus.cc:374] T a59c5115d4da4d179314104f1c793bab P ca52468b263a4d3283aca23c0771c185 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:19:17.104417 12202 raft_consensus.cc:729] T a59c5115d4da4d179314104f1c793bab P ca52468b263a4d3283aca23c0771c185 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca52468b263a4d3283aca23c0771c185, State: Initialized, Role: FOLLOWER
I20260928 05:19:17.105437 12202 consensus_queue.cc:260] T a59c5115d4da4d179314104f1c793bab P ca52468b263a4d3283aca23c0771c185 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca52468b263a4d3283aca23c0771c185" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40649 } } peers { permanent_uuid: "fd46289041554333896772e7fc1e0efd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40053 } } peers { permanent_uuid: "d8662becc5334b65bbef6990a7ac4a44" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45479 } }
I20260928 05:19:17.106690 12203 tablet_bootstrap.cc:492] T a59c5115d4da4d179314104f1c793bab P fd46289041554333896772e7fc1e0efd: No bootstrap required, opened a new log
I20260928 05:19:17.106974 12203 ts_tablet_manager.cc:1397] T a59c5115d4da4d179314104f1c793bab P fd46289041554333896772e7fc1e0efd: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.006s
I20260928 05:19:17.109244 12204 tablet_bootstrap.cc:492] T a59c5115d4da4d179314104f1c793bab P d8662becc5334b65bbef6990a7ac4a44: No bootstrap required, opened a new log
I20260928 05:19:17.109509 12204 ts_tablet_manager.cc:1397] T a59c5115d4da4d179314104f1c793bab P d8662becc5334b65bbef6990a7ac4a44: Time spent bootstrapping tablet: real 0.025s user 0.010s sys 0.006s
I20260928 05:19:17.111368 12202 ts_tablet_manager.cc:1428] T a59c5115d4da4d179314104f1c793bab P ca52468b263a4d3283aca23c0771c185: Time spent starting tablet: real 0.019s user 0.016s sys 0.000s
I20260928 05:19:17.117424 12203 raft_consensus.cc:348] T a59c5115d4da4d179314104f1c793bab P fd46289041554333896772e7fc1e0efd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca52468b263a4d3283aca23c0771c185" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40649 } } peers { permanent_uuid: "fd46289041554333896772e7fc1e0efd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40053 } } peers { permanent_uuid: "d8662becc5334b65bbef6990a7ac4a44" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45479 } }
I20260928 05:19:17.117769 12203 raft_consensus.cc:374] T a59c5115d4da4d179314104f1c793bab P fd46289041554333896772e7fc1e0efd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:19:17.117854 12203 raft_consensus.cc:729] T a59c5115d4da4d179314104f1c793bab P fd46289041554333896772e7fc1e0efd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd46289041554333896772e7fc1e0efd, State: Initialized, Role: FOLLOWER
I20260928 05:19:17.118541 12203 consensus_queue.cc:260] T a59c5115d4da4d179314104f1c793bab P fd46289041554333896772e7fc1e0efd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca52468b263a4d3283aca23c0771c185" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40649 } } peers { permanent_uuid: "fd46289041554333896772e7fc1e0efd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40053 } } peers { permanent_uuid: "d8662becc5334b65bbef6990a7ac4a44" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45479 } }
I20260928 05:19:17.119529 12049 heartbeater.cc:503] Master 127.0.97.62:36475 was elected leader, sending a full tablet report...
I20260928 05:19:17.119378 12204 raft_consensus.cc:348] T a59c5115d4da4d179314104f1c793bab P d8662becc5334b65bbef6990a7ac4a44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca52468b263a4d3283aca23c0771c185" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40649 } } peers { permanent_uuid: "fd46289041554333896772e7fc1e0efd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40053 } } peers { permanent_uuid: "d8662becc5334b65bbef6990a7ac4a44" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45479 } }
I20260928 05:19:17.119725 12204 raft_consensus.cc:374] T a59c5115d4da4d179314104f1c793bab P d8662becc5334b65bbef6990a7ac4a44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:19:17.119803 12204 raft_consensus.cc:729] T a59c5115d4da4d179314104f1c793bab P d8662becc5334b65bbef6990a7ac4a44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8662becc5334b65bbef6990a7ac4a44, State: Initialized, Role: FOLLOWER
I20260928 05:19:17.120200 12203 ts_tablet_manager.cc:1428] T a59c5115d4da4d179314104f1c793bab P fd46289041554333896772e7fc1e0efd: Time spent starting tablet: real 0.013s user 0.010s sys 0.004s
I20260928 05:19:17.120466 12204 consensus_queue.cc:260] T a59c5115d4da4d179314104f1c793bab P d8662becc5334b65bbef6990a7ac4a44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca52468b263a4d3283aca23c0771c185" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40649 } } peers { permanent_uuid: "fd46289041554333896772e7fc1e0efd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40053 } } peers { permanent_uuid: "d8662becc5334b65bbef6990a7ac4a44" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45479 } }
I20260928 05:19:17.121526 12183 heartbeater.cc:503] Master 127.0.97.62:36475 was elected leader, sending a full tablet report...
I20260928 05:19:17.122063 12204 ts_tablet_manager.cc:1428] T a59c5115d4da4d179314104f1c793bab P d8662becc5334b65bbef6990a7ac4a44: Time spent starting tablet: real 0.012s user 0.010s sys 0.002s
W20260928 05:19:17.152581 12050 tablet.cc:2369] T a59c5115d4da4d179314104f1c793bab P fd46289041554333896772e7fc1e0efd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 05:19:17.223415 11907 tablet.cc:2369] T a59c5115d4da4d179314104f1c793bab P ca52468b263a4d3283aca23c0771c185: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 05:19:17.224442 12184 tablet.cc:2369] T a59c5115d4da4d179314104f1c793bab P d8662becc5334b65bbef6990a7ac4a44: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 05:19:17.254798 12210 raft_consensus.cc:482] T a59c5115d4da4d179314104f1c793bab P d8662becc5334b65bbef6990a7ac4a44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:19:17.255064 12210 raft_consensus.cc:504] T a59c5115d4da4d179314104f1c793bab P d8662becc5334b65bbef6990a7ac4a44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca52468b263a4d3283aca23c0771c185" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40649 } } peers { permanent_uuid: "fd46289041554333896772e7fc1e0efd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40053 } } peers { permanent_uuid: "d8662becc5334b65bbef6990a7ac4a44" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45479 } }
I20260928 05:19:17.256662 12210 leader_election.cc:288] T a59c5115d4da4d179314104f1c793bab P d8662becc5334b65bbef6990a7ac4a44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ca52468b263a4d3283aca23c0771c185 (127.0.97.1:40649), fd46289041554333896772e7fc1e0efd (127.0.97.2:40053)
I20260928 05:19:17.262841 12186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:19:17.256835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40649 (local address 127.0.97.3:42545)
0928 05:19:17.257073 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0928 05:19:17.257091 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 05:19:17.257223 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:19:17.258045 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:19:17.258048 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:19:17.258071 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:19:17.258334 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:19:17.258345 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:19:17.259456 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:19:17.259461 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:19:17.261004 (+ 1543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:19:17.261015 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:19:17.262174 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:19:17.262180 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:19:17.262336 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:19:17.262374 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 05:19:17.262477 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 05:19:17.262582 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":61}
I20260928 05:19:17.262894 12197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:19:17.257552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42545 (local address 127.0.97.1:40649)
0928 05:19:17.257735 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 05:19:17.257740 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:19:17.257759 (+ 19us) server_negotiation.cc:407] Connection header received
0928 05:19:17.257827 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:19:17.257832 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:19:17.257905 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:19:17.257998 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:19:17.258470 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:19:17.259331 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:19:17.261144 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:19:17.262034 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:19:17.262126 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:19:17.262195 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 05:19:17.262522 (+ 327us) server_negotiation.cc:299] Negotiation successful
0928 05:19:17.262648 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 05:19:17.263101 12199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:19:17.257316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:32827 (local address 127.0.97.2:40053)
0928 05:19:17.257495 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 05:19:17.257500 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:19:17.258163 (+ 663us) server_negotiation.cc:407] Connection header received
0928 05:19:17.258244 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:19:17.258248 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:19:17.258330 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:19:17.258431 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:19:17.258890 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:19:17.259633 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:19:17.261391 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:19:17.262207 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:19:17.262332 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:19:17.262417 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 05:19:17.262747 (+ 330us) server_negotiation.cc:299] Negotiation successful
0928 05:19:17.262860 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 05:19:17.263438 12211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:19:17.257087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40053 (local address 127.0.97.3:32827)
0928 05:19:17.258015 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0928 05:19:17.258034 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 05:19:17.258140 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:19:17.258446 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:19:17.258449 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:19:17.258470 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:19:17.258761 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:19:17.258773 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:19:17.259786 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:19:17.259790 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:19:17.261251 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:19:17.261264 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:19:17.262380 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:19:17.262384 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:19:17.262591 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:19:17.262623 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 05:19:17.262713 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 05:19:17.262834 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":801,"thread_start_us":653,"threads_started":1}
I20260928 05:19:17.263854 12209 raft_consensus.cc:482] T a59c5115d4da4d179314104f1c793bab P fd46289041554333896772e7fc1e0efd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:19:17.263880 11861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a59c5115d4da4d179314104f1c793bab" candidate_uuid: "d8662becc5334b65bbef6990a7ac4a44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca52468b263a4d3283aca23c0771c185" is_pre_election: true
I20260928 05:19:17.264393 11861 raft_consensus.cc:2454] T a59c5115d4da4d179314104f1c793bab P ca52468b263a4d3283aca23c0771c185 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8662becc5334b65bbef6990a7ac4a44 in term 0.
I20260928 05:19:17.264117 12209 raft_consensus.cc:504] T a59c5115d4da4d179314104f1c793bab P fd46289041554333896772e7fc1e0efd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca52468b263a4d3283aca23c0771c185" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40649 } } peers { permanent_uuid: "fd46289041554333896772e7fc1e0efd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40053 } } peers { permanent_uuid: "d8662becc5334b65bbef6990a7ac4a44" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45479 } }
I20260928 05:19:17.265036 12073 leader_election.cc:302] T a59c5115d4da4d179314104f1c793bab P d8662becc5334b65bbef6990a7ac4a44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ca52468b263a4d3283aca23c0771c185, d8662becc5334b65bbef6990a7ac4a44; no voters:
I20260928 05:19:17.264915 12004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a59c5115d4da4d179314104f1c793bab" candidate_uuid: "d8662becc5334b65bbef6990a7ac4a44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd46289041554333896772e7fc1e0efd" is_pre_election: true
I20260928 05:19:17.265380 12210 raft_consensus.cc:2781] T a59c5115d4da4d179314104f1c793bab P d8662becc5334b65bbef6990a7ac4a44 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 05:19:17.265416 12004 raft_consensus.cc:2454] T a59c5115d4da4d179314104f1c793bab P fd46289041554333896772e7fc1e0efd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8662becc5334b65bbef6990a7ac4a44 in term 0.
I20260928 05:19:17.265533 12210 raft_consensus.cc:482] T a59c5115d4da4d179314104f1c793bab P d8662becc5334b65bbef6990a7ac4a44 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 05:19:17.265614 12210 raft_consensus.cc:3037] T a59c5115d4da4d179314104f1c793bab P d8662becc5334b65bbef6990a7ac4a44 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:19:17.266386 12209 leader_election.cc:288] T a59c5115d4da4d179314104f1c793bab P fd46289041554333896772e7fc1e0efd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ca52468b263a4d3283aca23c0771c185 (127.0.97.1:40649), d8662becc5334b65bbef6990a7ac4a44 (127.0.97.3:45479)
I20260928 05:19:17.268719 12210 raft_consensus.cc:504] T a59c5115d4da4d179314104f1c793bab P d8662becc5334b65bbef6990a7ac4a44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca52468b263a4d3283aca23c0771c185" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40649 } } peers { permanent_uuid: "fd46289041554333896772e7fc1e0efd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40053 } } peers { permanent_uuid: "d8662becc5334b65bbef6990a7ac4a44" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45479 } }
I20260928 05:19:17.269747 11861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a59c5115d4da4d179314104f1c793bab" candidate_uuid: "d8662becc5334b65bbef6990a7ac4a44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca52468b263a4d3283aca23c0771c185"
I20260928 05:19:17.269941 11861 raft_consensus.cc:3037] T a59c5115d4da4d179314104f1c793bab P ca52468b263a4d3283aca23c0771c185 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:19:17.270154 12004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a59c5115d4da4d179314104f1c793bab" candidate_uuid: "d8662becc5334b65bbef6990a7ac4a44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd46289041554333896772e7fc1e0efd"
I20260928 05:19:17.270345 12004 raft_consensus.cc:3037] T a59c5115d4da4d179314104f1c793bab P fd46289041554333896772e7fc1e0efd [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:19:17.273547 12004 raft_consensus.cc:2454] T a59c5115d4da4d179314104f1c793bab P fd46289041554333896772e7fc1e0efd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8662becc5334b65bbef6990a7ac4a44 in term 1.
I20260928 05:19:17.273584 11861 raft_consensus.cc:2454] T a59c5115d4da4d179314104f1c793bab P ca52468b263a4d3283aca23c0771c185 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8662becc5334b65bbef6990a7ac4a44 in term 1.
I20260928 05:19:17.273955 12071 leader_election.cc:302] T a59c5115d4da4d179314104f1c793bab P d8662becc5334b65bbef6990a7ac4a44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d8662becc5334b65bbef6990a7ac4a44, fd46289041554333896772e7fc1e0efd; no voters:
I20260928 05:19:17.274240 12201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:19:17.267181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48171 (local address 127.0.97.3:45479)
0928 05:19:17.267367 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 05:19:17.267374 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:19:17.267853 (+ 479us) server_negotiation.cc:407] Connection header received
0928 05:19:17.268022 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:19:17.268027 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:19:17.268076 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:19:17.268140 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:19:17.268606 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:19:17.269554 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:19:17.271985 (+ 2431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:19:17.273100 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:19:17.273183 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:19:17.273266 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 05:19:17.273898 (+ 632us) server_negotiation.cc:299] Negotiation successful
0928 05:19:17.274040 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 05:19:17.274250 12197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:19:17.266608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50999 (local address 127.0.97.1:40649)
0928 05:19:17.266778 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 05:19:17.266785 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:19:17.267170 (+ 385us) server_negotiation.cc:407] Connection header received
0928 05:19:17.267388 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:19:17.267393 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:19:17.267454 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:19:17.267511 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:19:17.268225 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:19:17.269191 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:19:17.271730 (+ 2539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:19:17.273044 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:19:17.273130 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:19:17.273225 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 05:19:17.273882 (+ 657us) server_negotiation.cc:299] Negotiation successful
0928 05:19:17.274013 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 05:19:17.274818 12213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:19:17.267108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45479 (local address 127.0.97.2:48171)
0928 05:19:17.267764 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0928 05:19:17.267780 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 05:19:17.267915 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:19:17.268163 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:19:17.268167 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:19:17.268189 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:19:17.268465 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:19:17.268473 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:19:17.269700 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:19:17.269704 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:19:17.271804 (+ 2100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:19:17.271818 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:19:17.273489 (+ 1671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:19:17.273495 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:19:17.273713 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:19:17.273760 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 05:19:17.274367 (+ 607us) client_negotiation.cc:240] Negotiation successful
0928 05:19:17.274490 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"mutex_wait_us":24,"spinlock_wait_cycles":768,"thread_start_us":398,"threads_started":1}
I20260928 05:19:17.274991 12212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:19:17.266536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40649 (local address 127.0.97.2:50999)
0928 05:19:17.267082 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0928 05:19:17.267099 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 05:19:17.267259 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:19:17.267693 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:19:17.267697 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:19:17.267721 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:19:17.268075 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:19:17.268086 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:19:17.269359 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:19:17.269364 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:19:17.271550 (+ 2186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:19:17.271564 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:19:17.273459 (+ 1895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:19:17.273468 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:19:17.273672 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:19:17.273750 (+ 78us) client_negotiation.cc:769] Sending connection context
0928 05:19:17.274390 (+ 640us) client_negotiation.cc:240] Negotiation successful
0928 05:19:17.274532 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"mutex_wait_us":44,"spinlock_wait_cycles":30720,"thread_start_us":256,"threads_started":1}
I20260928 05:19:17.275424 11861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a59c5115d4da4d179314104f1c793bab" candidate_uuid: "fd46289041554333896772e7fc1e0efd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca52468b263a4d3283aca23c0771c185" is_pre_election: true
I20260928 05:19:17.275655 11861 raft_consensus.cc:2379] T a59c5115d4da4d179314104f1c793bab P ca52468b263a4d3283aca23c0771c185 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fd46289041554333896772e7fc1e0efd in current term 1: Already voted for candidate d8662becc5334b65bbef6990a7ac4a44 in this term.
I20260928 05:19:17.276959 12210 leader_election.cc:288] T a59c5115d4da4d179314104f1c793bab P d8662becc5334b65bbef6990a7ac4a44 [CANDIDATE]: Term 1 election: Requested vote from peers ca52468b263a4d3283aca23c0771c185 (127.0.97.1:40649), fd46289041554333896772e7fc1e0efd (127.0.97.2:40053)
I20260928 05:19:17.277117 12210 raft_consensus.cc:2781] T a59c5115d4da4d179314104f1c793bab P d8662becc5334b65bbef6990a7ac4a44 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:19:17.277238 12210 raft_consensus.cc:686] T a59c5115d4da4d179314104f1c793bab P d8662becc5334b65bbef6990a7ac4a44 [term 1 LEADER]: Becoming Leader. State: Replica: d8662becc5334b65bbef6990a7ac4a44, State: Running, Role: LEADER
I20260928 05:19:17.277715 12210 consensus_queue.cc:237] T a59c5115d4da4d179314104f1c793bab P d8662becc5334b65bbef6990a7ac4a44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca52468b263a4d3283aca23c0771c185" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40649 } } peers { permanent_uuid: "fd46289041554333896772e7fc1e0efd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40053 } } peers { permanent_uuid: "d8662becc5334b65bbef6990a7ac4a44" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45479 } }
I20260928 05:19:17.277918 12138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a59c5115d4da4d179314104f1c793bab" candidate_uuid: "fd46289041554333896772e7fc1e0efd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8662becc5334b65bbef6990a7ac4a44" is_pre_election: true
I20260928 05:19:17.278765 11939 leader_election.cc:302] T a59c5115d4da4d179314104f1c793bab P fd46289041554333896772e7fc1e0efd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd46289041554333896772e7fc1e0efd; no voters: ca52468b263a4d3283aca23c0771c185, d8662becc5334b65bbef6990a7ac4a44
I20260928 05:19:17.279083 12209 raft_consensus.cc:2726] T a59c5115d4da4d179314104f1c793bab P fd46289041554333896772e7fc1e0efd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 05:19:17.281805 11693 catalog_manager.cc:5495] T a59c5115d4da4d179314104f1c793bab P d8662becc5334b65bbef6990a7ac4a44 reported cstate change: term changed from 0 to 1, leader changed from <none> to d8662becc5334b65bbef6990a7ac4a44 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "d8662becc5334b65bbef6990a7ac4a44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca52468b263a4d3283aca23c0771c185" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd46289041554333896772e7fc1e0efd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40053 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8662becc5334b65bbef6990a7ac4a44" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45479 } health_report { overall_health: HEALTHY } } }
I20260928 05:19:17.314333 12201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:19:17.309134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45666 (local address 127.0.97.3:45479)
0928 05:19:17.309306 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 05:19:17.309311 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:19:17.309366 (+ 55us) server_negotiation.cc:407] Connection header received
0928 05:19:17.309580 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:19:17.309584 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:19:17.309664 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:19:17.309789 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:19:17.310145 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:19:17.310848 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:19:17.312271 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:19:17.312668 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:19:17.312778 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:19:17.313294 (+ 516us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:19:17.313765 (+ 471us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:19:17.313889 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 05:19:17.313997 (+ 108us) server_negotiation.cc:299] Negotiation successful
0928 05:19:17.314076 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 05:19:17.321512 12004 raft_consensus.cc:1261] T a59c5115d4da4d179314104f1c793bab P fd46289041554333896772e7fc1e0efd [term 1 FOLLOWER]: Refusing update from remote peer d8662becc5334b65bbef6990a7ac4a44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:19:17.321511 11861 raft_consensus.cc:1261] T a59c5115d4da4d179314104f1c793bab P ca52468b263a4d3283aca23c0771c185 [term 1 FOLLOWER]: Refusing update from remote peer d8662becc5334b65bbef6990a7ac4a44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:19:17.322253 12214 consensus_queue.cc:1035] T a59c5115d4da4d179314104f1c793bab P d8662becc5334b65bbef6990a7ac4a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca52468b263a4d3283aca23c0771c185" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:19:17.322597 12210 consensus_queue.cc:1035] T a59c5115d4da4d179314104f1c793bab P d8662becc5334b65bbef6990a7ac4a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd46289041554333896772e7fc1e0efd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:19:17.335577 12219 mvcc.cc:205] Tried to move back new op lower bound from 7334186013980278784 to 7334186013813243904. Current Snapshot: MvccSnapshot[applied={T|T < 7334186013980278784}]
I20260928 05:19:17.346050 11692 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58092:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:19:17.346288 11692 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58092:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:19:17.349638 11692 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 32151fb09f4a40d3a227e8f053b7cf59: Sending DeleteTablet for 3 replicas of tablet a59c5115d4da4d179314104f1c793bab
I20260928 05:19:17.350607 12118 tablet_service.cc:1514] Processing DeleteTablet for tablet a59c5115d4da4d179314104f1c793bab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:19:17 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:45654
I20260928 05:19:17.350728 11841 tablet_service.cc:1514] Processing DeleteTablet for tablet a59c5115d4da4d179314104f1c793bab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:19:17 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39978
I20260928 05:19:17.351090 11984 tablet_service.cc:1514] Processing DeleteTablet for tablet a59c5115d4da4d179314104f1c793bab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:19:17 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41228
I20260928 05:19:17.354231 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 11773
I20260928 05:19:17.363844 12231 tablet_replica.cc:330] T a59c5115d4da4d179314104f1c793bab P fd46289041554333896772e7fc1e0efd: stopping tablet replica
I20260928 05:19:17.364305 12231 raft_consensus.cc:2229] T a59c5115d4da4d179314104f1c793bab P fd46289041554333896772e7fc1e0efd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:19:17.368551 12231 raft_consensus.cc:2258] T a59c5115d4da4d179314104f1c793bab P fd46289041554333896772e7fc1e0efd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:19:17.370860 12231 ts_tablet_manager.cc:1905] T a59c5115d4da4d179314104f1c793bab P fd46289041554333896772e7fc1e0efd: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 28 05:19:17 dist-test-slave-whsn krb5kdc[11217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572753, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260928 05:19:17.384673 12231 ts_tablet_manager.cc:1918] T a59c5115d4da4d179314104f1c793bab P fd46289041554333896772e7fc1e0efd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:19:17.385527 12229 tablet_replica.cc:330] T a59c5115d4da4d179314104f1c793bab P ca52468b263a4d3283aca23c0771c185: stopping tablet replica
I20260928 05:19:17.386247 12230 tablet_replica.cc:330] T a59c5115d4da4d179314104f1c793bab P d8662becc5334b65bbef6990a7ac4a44: stopping tablet replica
I20260928 05:19:17.386629 12230 raft_consensus.cc:2229] T a59c5115d4da4d179314104f1c793bab P d8662becc5334b65bbef6990a7ac4a44 [term 1 LEADER]: Raft consensus shutting down.
I20260928 05:19:17.387181 12230 raft_consensus.cc:2258] T a59c5115d4da4d179314104f1c793bab P d8662becc5334b65bbef6990a7ac4a44 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:19:17.388787 12229 raft_consensus.cc:2229] T a59c5115d4da4d179314104f1c793bab P ca52468b263a4d3283aca23c0771c185 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:19:17.389390 12229 raft_consensus.cc:2258] T a59c5115d4da4d179314104f1c793bab P ca52468b263a4d3283aca23c0771c185 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:19:17.391495 12231 log.cc:1196] T a59c5115d4da4d179314104f1c793bab P fd46289041554333896772e7fc1e0efd: Deleting WAL directory at /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/minicluster-data/ts-1/wal/wals/a59c5115d4da4d179314104f1c793bab
I20260928 05:19:17.392016 12231 ts_tablet_manager.cc:1939] T a59c5115d4da4d179314104f1c793bab P fd46289041554333896772e7fc1e0efd: Deleting consensus metadata
I20260928 05:19:17.393399 12229 ts_tablet_manager.cc:1905] T a59c5115d4da4d179314104f1c793bab P ca52468b263a4d3283aca23c0771c185: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:19:17.393909 11677 catalog_manager.cc:4841] TS fd46289041554333896772e7fc1e0efd (127.0.97.2:40053): tablet a59c5115d4da4d179314104f1c793bab (table test-table [id=647e3abe038b49ef8b4636dcbf35906e]) successfully deleted
I20260928 05:19:17.396237 12197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:19:17.355745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39980 (local address 127.0.97.1:40649)
0928 05:19:17.355917 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 05:19:17.355924 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:19:17.359744 (+ 3820us) server_negotiation.cc:407] Connection header received
0928 05:19:17.360587 (+ 843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:19:17.360594 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:19:17.360688 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:19:17.360760 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:19:17.365409 (+ 4649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:19:17.366623 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:19:17.371284 (+ 4661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:19:17.371991 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:19:17.372090 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:19:17.388515 (+ 16425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:19:17.388527 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:19:17.388544 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:19:17.388598 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:19:17.392318 (+ 3720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:19:17.393411 (+ 1093us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:19:17.393431 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:19:17.393435 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:19:17.393589 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:19:17.394054 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:19:17.394060 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:19:17.394063 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:19:17.394527 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:19:17.394796 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0928 05:19:17.395430 (+ 634us) server_negotiation.cc:299] Negotiation successful
0928 05:19:17.395713 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 05:19:17.397660 11901 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:19:17.397678 12230 ts_tablet_manager.cc:1905] T a59c5115d4da4d179314104f1c793bab P d8662becc5334b65bbef6990a7ac4a44: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:19:17.405517 12230 ts_tablet_manager.cc:1918] T a59c5115d4da4d179314104f1c793bab P d8662becc5334b65bbef6990a7ac4a44: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:19:17.405678 12230 log.cc:1196] T a59c5115d4da4d179314104f1c793bab P d8662becc5334b65bbef6990a7ac4a44: Deleting WAL directory at /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572622576160-388-0/minicluster-data/ts-2/wal/wals/a59c5115d4da4d179314104f1c793bab
I20260928 05:19:17.406533 12230 ts_tablet_manager.cc:1939] T a59c5115d4da4d179314104f1c793bab P d8662becc5334b65bbef6990a7ac4a44: Deleting consensus metadata
I20260928 05:19:17.408300 11679 catalog_manager.cc:4841] TS d8662becc5334b65bbef6990a7ac4a44 (127.0.97.3:45479): tablet a59c5115d4da4d179314104f1c793bab (table test-table [id=647e3abe038b49ef8b4636dcbf35906e]) successfully deleted
I20260928 05:19:17.486553 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 11773
W20260928 05:19:17.506875 11679 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:40649 (error 108)
I20260928 05:19:17.506945 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 11912
I20260928 05:19:17.509105 12200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:19:17.508701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40649 (local address 127.0.0.1:39988)
0928 05:19:17.508870 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 05:19:17.508973 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260928 05:19:17.509408 11679 catalog_manager.cc:4584] TS ca52468b263a4d3283aca23c0771c185 (127.0.97.1:40649): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a59c5115d4da4d179314104f1c793bab: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40649: connect: Connection refused (error 111)
Sep 28 05:19:17 dist-test-slave-whsn krb5kdc[11217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572753, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260928 05:19:17.525000 12199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:19:17.507760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41244 (local address 127.0.97.2:40053)
0928 05:19:17.508350 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0928 05:19:17.508356 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:19:17.508374 (+ 18us) server_negotiation.cc:407] Connection header received
0928 05:19:17.508447 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:19:17.508453 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:19:17.508541 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:19:17.508633 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:19:17.510359 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:19:17.511080 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:19:17.513064 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:19:17.513458 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:19:17.513549 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:19:17.518701 (+ 5152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:19:17.518719 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:19:17.518733 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:19:17.518766 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:19:17.522067 (+ 3301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:19:17.522770 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:19:17.522780 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:19:17.522783 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:19:17.522932 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:19:17.523289 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:19:17.523294 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:19:17.523297 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:19:17.523815 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:19:17.524104 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0928 05:19:17.524366 (+ 262us) server_negotiation.cc:299] Negotiation successful
0928 05:19:17.524593 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462}
I20260928 05:19:17.525635 12044 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:19:17.563066 12200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:19:17.562562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40649 (local address 127.0.0.1:39992)
0928 05:19:17.562779 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0928 05:19:17.562882 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260928 05:19:17.564167 12200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:19:17.563772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40649 (local address 127.0.0.1:39994)
0928 05:19:17.563926 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 05:19:17.564013 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260928 05:19:17.615624 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 11912
I20260928 05:19:17.624754 12200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:19:17.624201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40649 (local address 127.0.0.1:39998)
0928 05:19:17.624424 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0928 05:19:17.624535 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260928 05:19:17.625908 12200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:19:17.625527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40649 (local address 127.0.0.1:40000)
0928 05:19:17.625687 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 05:19:17.625770 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260928 05:19:17.629436 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 12055
Sep 28 05:19:17 dist-test-slave-whsn krb5kdc[11217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572753, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260928 05:19:17.645239 12201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:19:17.630128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45676 (local address 127.0.97.3:45479)
0928 05:19:17.630297 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 05:19:17.630303 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:19:17.630325 (+ 22us) server_negotiation.cc:407] Connection header received
0928 05:19:17.630525 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:19:17.630531 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:19:17.630624 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:19:17.630743 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:19:17.631892 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:19:17.632994 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:19:17.634205 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:19:17.634867 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:19:17.635010 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:19:17.639526 (+ 4516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:19:17.639537 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:19:17.639551 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:19:17.639581 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:19:17.642570 (+ 2989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:19:17.643308 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:19:17.643314 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:19:17.643317 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:19:17.643426 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:19:17.643825 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:19:17.643828 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:19:17.643830 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:19:17.644210 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:19:17.644512 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0928 05:19:17.644809 (+ 297us) server_negotiation.cc:299] Negotiation successful
0928 05:19:17.644991 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 05:19:17.646008 12178 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:19:17.706911 12200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:19:17.706361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40649 (local address 127.0.0.1:40014)
0928 05:19:17.706602 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0928 05:19:17.706714 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260928 05:19:17.708047 12200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:19:17.707619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40649 (local address 127.0.0.1:40030)
0928 05:19:17.707821 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0928 05:19:17.707914 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260928 05:19:17.764461 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 12055
I20260928 05:19:17.779304 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 11661
I20260928 05:19:17.780504 11723 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:19:17.887290 12200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:19:17.885841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40649 (local address 127.0.0.1:40042)
0928 05:19:17.886878 (+ 1037us) negotiation.cc:107] Waiting for socket to connect
0928 05:19:17.886996 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":884}
I20260928 05:19:17.887565 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 11661
I20260928 05:19:17.907567 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 05:19:18.241657 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 05:19:18.241847 388 mini_ranger.cc:66] Stopping Ranger...
I20260928 05:19:18.575848 388 mini_ranger.cc:68] Stopped Ranger
2026-09-28 05:19:18.576 UTC [11247] LOG: received smart shutdown request
2026-09-28 05:19:18.579 UTC [11247] LOG: background worker "logical replication launcher" (PID 11256) exited with exit code 1
2026-09-28 05:19:18.579 UTC [11251] LOG: shutting down
2026-09-28 05:19:18.715 UTC [11247] LOG: database system is shut down
2026-09-28T05:19:18Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (46813 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:19:18 dist-test-slave-whsn krb5kdc[12246](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:19:18 dist-test-slave-whsn krb5kdc[12246](info): set up 2 sockets
Sep 28 05:19:18 dist-test-slave-whsn krb5kdc[12246](info): commencing operation
krb5kdc: starting...
W20260928 05:19:19.812781 388 mini_kdc.cc:121] Time spent starting KDC: real 0.961s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/krb5kdc/test-user.keytab.
Sep 28 05:19:19 dist-test-slave-whsn krb5kdc[12246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:19:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:19:19Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260928 05:19:20.083592 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-tasku1htJC/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/postgres -l logfile start
2026-09-28 05:19:22.273 UTC [12276] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 05:19:22.273 UTC [12276] LOG: listening on IPv4 address "127.0.97.20", port 51115
2026-09-28 05:19:22.276 UTC [12276] LOG: listening on Unix socket "/tmp/.s.PGSQL.51115"
2026-09-28 05:19:22.295 UTC [12279] LOG: database system was shut down at 2026-09-28 05:19:21 UTC
2026-09-28 05:19:22.298 UTC [12276] LOG: database system is ready to accept connections
I20260928 05:19:23.188575 388 mini_postgres.cc:96] Postgres bound to 51115
2026-09-28 05:19:23.203 UTC [12287] FATAL: database "slave" does not exist
127.0.97.20:51115 - accepting connections
I20260928 05:19:23.204214 388 mini_ranger.cc:161] Starting Ranger...
I20260928 05:19:23.226329 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 05:19:23.626349 388 mini_ranger.cc:87] Created ranger Postgres database
I20260928 05:19:23.626439 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ranger-admin
2026-09-28 05:19:23,897 [I] DB FLAVOR :POSTGRES
2026-09-28 05:19:23,898 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 05:19:23,898 [I] Checking connection..
2026-09-28 05:19:23,898 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasku1htJC/build/asan/bin/postgresql.jar:/tmp/dist-test-tasku1htJC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:51115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 05:19:24,154 [I] Checking connection passed.
2026-09-28 05:19:24,154 [I] --------- Verifying version history table ---------
2026-09-28 05:19:24,154 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasku1htJC/build/asan/bin/postgresql.jar:/tmp/dist-test-tasku1htJC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:51115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 05:19:24,387 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 05:19:24,387 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasku1htJC/build/asan/bin/postgresql.jar:/tmp/dist-test-tasku1htJC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:51115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 05:19:24,631 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 05:19:24,632 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 05:19:24,632 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasku1htJC/build/asan/bin/postgresql.jar:/tmp/dist-test-tasku1htJC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:51115/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasku1htJC/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 05:19:24.872 UTC [12351] WARNING: there is no transaction in progress
2026-09-28 05:19:24,877 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 05:19:24,877 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasku1htJC/build/asan/bin/postgresql.jar:/tmp/dist-test-tasku1htJC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:51115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 05:19:25,117 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 05:19:25,117 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 05:19:25,117 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasku1htJC/build/asan/bin/postgresql.jar:/tmp/dist-test-tasku1htJC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:51115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 05:19:25,363 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasku1htJC/build/asan/bin/postgresql.jar:/tmp/dist-test-tasku1htJC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:51115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 05:19:25,617 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasku1htJC/build/asan/bin/postgresql.jar:/tmp/dist-test-tasku1htJC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:51115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-whsn.c.gcp-upstream.internal','N') ;"
2026-09-28 05:19:25,854 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasku1htJC/build/asan/bin/postgresql.jar:/tmp/dist-test-tasku1htJC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:51115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 05:19:26,099 [I] Table x_portal_user does not exist in database ranger
2026-09-28 05:19:26,099 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasku1htJC/build/asan/bin/postgresql.jar:/tmp/dist-test-tasku1htJC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:51115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 05:19:26,345 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 05:19:26,346 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasku1htJC/build/asan/bin/postgresql.jar:/tmp/dist-test-tasku1htJC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:51115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 05:19:26,606 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasku1htJC/build/asan/bin/postgresql.jar:/tmp/dist-test-tasku1htJC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:51115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 05:19:26,847 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 05:19:26,848 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasku1htJC/build/asan/bin/postgresql.jar:/tmp/dist-test-tasku1htJC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:51115/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasku1htJC/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 05:19:27.673 UTC [12459] WARNING: there is no transaction in progress
2026-09-28 05:19:27.688 UTC [12459] WARNING: there is no transaction in progress
2026-09-28 05:19:27.703 UTC [12459] WARNING: there is no transaction in progress
2026-09-28 05:19:27.716 UTC [12459] WARNING: there is no transaction in progress
2026-09-28 05:19:27.731 UTC [12459] WARNING: there is no transaction in progress
2026-09-28 05:19:27.746 UTC [12459] WARNING: there is no transaction in progress
2026-09-28 05:19:27.795 UTC [12459] WARNING: there is no transaction in progress
2026-09-28 05:19:27.800 UTC [12459] WARNING: there is no transaction in progress
2026-09-28 05:19:27.813 UTC [12459] WARNING: there is no transaction in progress
2026-09-28 05:19:27.822 UTC [12459] WARNING: there is no transaction in progress
2026-09-28 05:19:27.830 UTC [12459] WARNING: there is no transaction in progress
2026-09-28 05:19:27.842 UTC [12459] WARNING: there is no transaction in progress
2026-09-28 05:19:27.850 UTC [12459] WARNING: there is no transaction in progress
2026-09-28 05:19:27.856 UTC [12459] WARNING: there is no transaction in progress
2026-09-28 05:19:27.861 UTC [12459] WARNING: there is no transaction in progress
2026-09-28 05:19:27.866 UTC [12459] WARNING: there is no transaction in progress
2026-09-28 05:19:28,386 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 05:19:28,386 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasku1htJC/build/asan/bin/postgresql.jar:/tmp/dist-test-tasku1htJC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:51115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 05:19:28,619 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 05:19:28,619 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasku1htJC/build/asan/bin/postgresql.jar:/tmp/dist-test-tasku1htJC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:51115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 05:19:28,862 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 05:19:28,862 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasku1htJC/build/asan/bin/postgresql.jar:/tmp/dist-test-tasku1htJC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:51115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 05:19:29,112 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 05:19:29,112 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasku1htJC/build/asan/bin/postgresql.jar:/tmp/dist-test-tasku1htJC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:51115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 05:19:29,355 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasku1htJC/build/asan/bin/postgresql.jar:/tmp/dist-test-tasku1htJC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:51115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 05:19:29,586 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasku1htJC/build/asan/bin/postgresql.jar:/tmp/dist-test-tasku1htJC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:51115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-whsn.c.gcp-upstream.internal';"
2026-09-28 05:19:29,824 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 05:19:29,825 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasku1htJC/build/asan/bin/postgresql.jar:/tmp/dist-test-tasku1htJC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:51115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 05:19:30,067 [I] DB_PATCHES have already been applied
I20260928 05:19:30.072978 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ranger-admin:/tmp/dist-test-tasku1htJC/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-tasku1htJC/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-tasku1htJC/build/asan/bin/postgresql.jar:/tmp/dist-test-tasku1htJC/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 05:19:30.073055 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260928 05:19:30.085529 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:57983
Sep 28, 2026 5:19:30 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 5:19:30 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 5:19:30 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 5:19:30 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 5:19:30 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 5:19:31 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 5:19:31 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 05:19:31.131889 388 mini_ranger.cc:160] Time spent starting Ranger: real 7.928s user 0.000s sys 0.033s
Sep 28, 2026 5:19:50 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 5:19:50 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 28, 2026 5:19:50 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 5:19:50 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 5:19:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 5:19:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 5:19:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 5:19:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 5:19:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 5:19:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 5:19:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 5:19:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 5:19:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 5:19:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 5:19:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 5:19:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 5:19:50 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 5:19:51 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 05:19:51.740952 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 05:19:51 dist-test-slave-whsn krb5kdc[12246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572791, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 05:19:51.890982 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 05:19:51.910447 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 05:19:52.663331 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 05:19:52.663429 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ranger-kms
I20260928 05:19:52.663466 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:51115
2026-09-28 05:19:52,975 [I] DB FLAVOR :POSTGRES
2026-09-28 05:19:52,975 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 05:19:52,975 [I] Checking connection
2026-09-28 05:19:52,975 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasku1htJC/build/asan/bin/postgresql.jar:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:51115/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 05:19:53,193 [I] connection success
2026-09-28 05:19:53,193 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 05:19:53,193 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 05:19:53,193 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasku1htJC/build/asan/bin/postgresql.jar:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:51115/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 05:19:53,412 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 05:19:53,412 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 05:19:53,412 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 05:19:53,412 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasku1htJC/build/asan/bin/postgresql.jar:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:51115/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 05:19:53,682 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 05:19:54.106931 388 mini_ranger_kms.cc:310] Created kms service
I20260928 05:19:54.295826 388 mini_ranger_kms.cc:326] Created kudu user
I20260928 05:19:54.340583 388 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 05:19:54 dist-test-slave-whsn krb5kdc[12246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572791, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260928 05:19:54.702271 388 mini_ranger_kms.cc:384] Added ranger policy
I20260928 05:19:54.702517 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ranger-kms:/tmp/dist-test-tasku1htJC/build/asan/bin/postgresql.jar:/tmp/dist-test-tasku1htJC/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-tasku1htJC/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 05:19:54.702555 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260928 05:19:54.715054 388 mini_ranger_kms.cc:276] Ranger KMS PID: 12639
I20260928 05:19:54.715169 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:47701
Sep 28, 2026 5:19:55 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 5:19:55 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 5:19:55 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 5:19:55 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 5:19:56 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 5:19:56 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 05:19:56.720166 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.829s user 0.056s sys 0.270s
I20260928 05:19:56.720409 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 05:19:56.720566 388 mini_ranger_kms.cc:401] 127.0.97.20:47701/kms/v1/keys
Sep 28 05:19:56 dist-test-slave-whsn krb5kdc[12246](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790572796, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 05:19:57 dist-test-slave-whsn krb5kdc[12246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572796, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 28 05:19:57 dist-test-slave-whsn krb5kdc[12246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572796, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 28 05:19:59 dist-test-slave-whsn krb5kdc[12246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572799, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 05:19:59.819135 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37687
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37687
--ranger_config_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:47701/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:20:00.119483 12690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:20:00.119769 12690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:20:00.119894 12690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:20:00.127081 12690 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:20:00.127202 12690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:20:00.127282 12690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:20:00.127365 12690 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:20:00.127431 12690 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:20:00.137384 12690 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:47701/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37687
--ranger_config_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:20:00.138674 12690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:20:00.139947 12690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:20:00.152944 12696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:20:00.152969 12695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:20:00.153002 12698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:20:00.154117 12690 server_base.cc:1034] running on GCE node
I20260928 05:20:00.154822 12690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:20:00.156246 12690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:20:00.157437 12690 hybrid_clock.cc:648] HybridClock initialized: now 1790572800157382 us; error 53 us; skew 500 ppm
I20260928 05:20:00.157755 12690 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:20:00 dist-test-slave-whsn krb5kdc[12246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572800, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:20:00.164458 12690 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 05:20:00.165915 12690 webserver.cc:460] Webserver started at http://127.0.97.62:46749/ using document root <none> and password file <none>
I20260928 05:20:00.166517 12690 fs_manager.cc:359] Metadata directory not provided
I20260928 05:20:00.166615 12690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:20:00.166888 12690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 05:20:00 dist-test-slave-whsn krb5kdc[12246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572800, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260928 05:20:00.227666 12690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/master-0/data/instance:
uuid: "30e4edfa8186490b8cb4258d9524eae7"
format_stamp: "Formatted at 2026-09-28 05:20:00 on dist-test-slave-whsn"
server_key: "f8f9f02bebbb1cf53fdf44903a886d96"
server_key_iv: "fd0a719e426dad0385eef8dcb46637d6"
server_key_version: "kuduclusterkey@0"
I20260928 05:20:00.228469 12690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/master-0/wal/instance:
uuid: "30e4edfa8186490b8cb4258d9524eae7"
format_stamp: "Formatted at 2026-09-28 05:20:00 on dist-test-slave-whsn"
server_key: "f8f9f02bebbb1cf53fdf44903a886d96"
server_key_iv: "fd0a719e426dad0385eef8dcb46637d6"
server_key_version: "kuduclusterkey@0"
I20260928 05:20:00.233685 12690 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260928 05:20:00.248943 12705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:20:00.250393 12690 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 05:20:00.250547 12690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/master-0/wal
uuid: "30e4edfa8186490b8cb4258d9524eae7"
format_stamp: "Formatted at 2026-09-28 05:20:00 on dist-test-slave-whsn"
server_key: "f8f9f02bebbb1cf53fdf44903a886d96"
server_key_iv: "fd0a719e426dad0385eef8dcb46637d6"
server_key_version: "kuduclusterkey@0"
I20260928 05:20:00.250746 12690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:20:00.273023 12690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:20:00.276657 12690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:20:00.276963 12690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:20:00.294552 12690 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37687
I20260928 05:20:00.294569 12756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37687 every 8 connection(s)
I20260928 05:20:00.295827 12690 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:20:00.299448 12757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:20:00.300040 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 12690
I20260928 05:20:00.300261 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/master-0/wal/instance
I20260928 05:20:00.301069 388 external_mini_cluster.cc:1428] Setting key d2d3da01c19136df15f56eba10a247bc
I20260928 05:20:00.314429 12757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30e4edfa8186490b8cb4258d9524eae7: Bootstrap starting.
I20260928 05:20:00.318588 12757 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 30e4edfa8186490b8cb4258d9524eae7: Neither blocks nor log segments found. Creating new log.
I20260928 05:20:00.320016 12757 log.cc:824] T 00000000000000000000000000000000 P 30e4edfa8186490b8cb4258d9524eae7: Log is configured to *not* fsync() on all Append() calls
I20260928 05:20:00.323585 12757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30e4edfa8186490b8cb4258d9524eae7: No bootstrap required, opened a new log
I20260928 05:20:00.331947 12757 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30e4edfa8186490b8cb4258d9524eae7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30e4edfa8186490b8cb4258d9524eae7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37687 } }
I20260928 05:20:00.332448 12757 raft_consensus.cc:374] T 00000000000000000000000000000000 P 30e4edfa8186490b8cb4258d9524eae7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:20:00.332536 12757 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30e4edfa8186490b8cb4258d9524eae7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30e4edfa8186490b8cb4258d9524eae7, State: Initialized, Role: FOLLOWER
I20260928 05:20:00.333132 12757 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30e4edfa8186490b8cb4258d9524eae7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30e4edfa8186490b8cb4258d9524eae7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37687 } }
I20260928 05:20:00.333343 12757 raft_consensus.cc:388] T 00000000000000000000000000000000 P 30e4edfa8186490b8cb4258d9524eae7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:20:00.333488 12757 raft_consensus.cc:482] T 00000000000000000000000000000000 P 30e4edfa8186490b8cb4258d9524eae7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:20:00.333634 12757 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30e4edfa8186490b8cb4258d9524eae7 [term 0 FOLLOWER]: Advancing to term 1
Sep 28 05:20:00 dist-test-slave-whsn krb5kdc[12246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572799, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:20:00.336134 12757 raft_consensus.cc:504] T 00000000000000000000000000000000 P 30e4edfa8186490b8cb4258d9524eae7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30e4edfa8186490b8cb4258d9524eae7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37687 } }
I20260928 05:20:00.336640 12757 leader_election.cc:302] T 00000000000000000000000000000000 P 30e4edfa8186490b8cb4258d9524eae7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30e4edfa8186490b8cb4258d9524eae7; no voters:
I20260928 05:20:00.337227 12757 leader_election.cc:288] T 00000000000000000000000000000000 P 30e4edfa8186490b8cb4258d9524eae7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:20:00.337309 12762 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 30e4edfa8186490b8cb4258d9524eae7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:20:00.341403 12762 raft_consensus.cc:686] T 00000000000000000000000000000000 P 30e4edfa8186490b8cb4258d9524eae7 [term 1 LEADER]: Becoming Leader. State: Replica: 30e4edfa8186490b8cb4258d9524eae7, State: Running, Role: LEADER
I20260928 05:20:00.342044 12762 consensus_queue.cc:237] T 00000000000000000000000000000000 P 30e4edfa8186490b8cb4258d9524eae7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30e4edfa8186490b8cb4258d9524eae7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37687 } }
I20260928 05:20:00.345341 12760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:00.303153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58448 (local address 127.0.97.62:37687)
0928 05:20:00.321448 (+ 18295us) server_negotiation.cc:206] Beginning negotiation
0928 05:20:00.321466 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 05:20:00.322048 (+ 582us) server_negotiation.cc:407] Connection header received
0928 05:20:00.324495 (+ 2447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:20:00.324524 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:20:00.325012 (+ 488us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:20:00.325424 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:20:00.326897 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:00.328839 (+ 1942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:00.330646 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:00.331694 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:00.331882 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:00.336349 (+ 4467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:20:00.336358 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:20:00.336376 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:20:00.336416 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:20:00.340009 (+ 3593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:00.340794 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:00.340801 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:00.340809 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:00.340996 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:00.341432 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:00.341436 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:00.341439 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:00.342053 (+ 614us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:20:00.343296 (+ 1243us) server_negotiation.cc:1035] Waiting for connection context
0928 05:20:00.343555 (+ 259us) server_negotiation.cc:299] Negotiation successful
0928 05:20:00.343922 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18002,"thread_start_us":17746,"threads_started":1}
I20260928 05:20:00.351152 12757 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30e4edfa8186490b8cb4258d9524eae7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:20:00.355420 12764 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30e4edfa8186490b8cb4258d9524eae7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "30e4edfa8186490b8cb4258d9524eae7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30e4edfa8186490b8cb4258d9524eae7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37687 } } }
I20260928 05:20:00.357934 12764 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30e4edfa8186490b8cb4258d9524eae7 [sys.catalog]: This master's current role is: LEADER
I20260928 05:20:00.358737 12769 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:20:00.361032 12763 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30e4edfa8186490b8cb4258d9524eae7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30e4edfa8186490b8cb4258d9524eae7. Latest consensus state: current_term: 1 leader_uuid: "30e4edfa8186490b8cb4258d9524eae7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30e4edfa8186490b8cb4258d9524eae7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37687 } } }
I20260928 05:20:00.362380 12763 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30e4edfa8186490b8cb4258d9524eae7 [sys.catalog]: This master's current role is: LEADER
I20260928 05:20:00.369786 12769 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:20:00.379017 12769 catalog_manager.cc:1347] Generated new cluster ID: 6eedfb183f36483d9878dc64bc8b5bf9
I20260928 05:20:00.379109 12769 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:20:00.380842 12757 ranger_client.cc:318] Using new properties file: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 05:20:00.401294 12769 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:20:00.403949 12769 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:20:00.433030 12769 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 30e4edfa8186490b8cb4258d9524eae7: Generated new TSK 0
I20260928 05:20:00.434175 12769 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260928 05:20:01.162029 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:47701/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37687
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 05:20:01 dist-test-slave-whsn krb5kdc[12246](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790572801, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260928 05:20:01.519976 12802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:20:01.520421 12802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:20:01.520612 12802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:20:01.529301 12802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:20:01.529397 12802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:20:01.529529 12802 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 05:20:01.539256 12802 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:47701/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37687
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:20:01.540426 12802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:20:01.541683 12802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:20:01.557029 12814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:20:01.557060 12812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:20:01.557776 12811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:20:01.562740 12802 server_base.cc:1034] running on GCE node
I20260928 05:20:01.563323 12802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:20:01.564309 12802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:20:01.565498 12802 hybrid_clock.cc:648] HybridClock initialized: now 1790572801565441 us; error 52 us; skew 500 ppm
I20260928 05:20:01.565804 12802 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:20:01 dist-test-slave-whsn krb5kdc[12246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572801, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:20:01.568719 12802 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260928 05:20:01.584944 12802 webserver.cc:460] Webserver started at http://127.0.97.1:41437/ using document root <none> and password file <none>
I20260928 05:20:01.585893 12802 fs_manager.cc:359] Metadata directory not provided
I20260928 05:20:01.586081 12802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:20:01.586450 12802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 05:20:01 dist-test-slave-whsn krb5kdc[12246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572801, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260928 05:20:01.605829 12802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ts-0/data/instance:
uuid: "52b3d46fb965452c9ca57b20aad37214"
format_stamp: "Formatted at 2026-09-28 05:20:01 on dist-test-slave-whsn"
server_key: "4968edef52267842c187378b2f5bccaf"
server_key_iv: "a9879d92b042eb9570b901041b1feebe"
server_key_version: "kuduclusterkey@0"
I20260928 05:20:01.607847 12802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ts-0/wal/instance:
uuid: "52b3d46fb965452c9ca57b20aad37214"
format_stamp: "Formatted at 2026-09-28 05:20:01 on dist-test-slave-whsn"
server_key: "4968edef52267842c187378b2f5bccaf"
server_key_iv: "a9879d92b042eb9570b901041b1feebe"
server_key_version: "kuduclusterkey@0"
I20260928 05:20:01.619977 12802 fs_manager.cc:691] Time spent creating directory manager: real 0.012s user 0.003s sys 0.008s
I20260928 05:20:01.635475 12822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:20:01.637261 12802 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 05:20:01.637506 12802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ts-0/wal
uuid: "52b3d46fb965452c9ca57b20aad37214"
format_stamp: "Formatted at 2026-09-28 05:20:01 on dist-test-slave-whsn"
server_key: "4968edef52267842c187378b2f5bccaf"
server_key_iv: "a9879d92b042eb9570b901041b1feebe"
server_key_version: "kuduclusterkey@0"
I20260928 05:20:01.637831 12802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:20:01.672456 12802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:20:01.676592 12802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:20:01.676899 12802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:20:01.677858 12802 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:20:01.679226 12802 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:20:01.679370 12802 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:20:01.679497 12802 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:20:01.679557 12802 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:20:01.723423 12802 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42651
I20260928 05:20:01.723438 12934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42651 every 8 connection(s)
I20260928 05:20:01.724750 12802 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ts-0/data/info.pb
I20260928 05:20:01.730384 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 12802
I20260928 05:20:01.730597 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ts-0/wal/instance
I20260928 05:20:01.731393 388 external_mini_cluster.cc:1428] Setting key 6342c7c5780c5268ebad1da10571e685
Sep 28 05:20:01 dist-test-slave-whsn krb5kdc[12246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572801, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:20:01.758432 12937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:01.727590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58769 (local address 127.0.97.62:37687)
0928 05:20:01.737210 (+ 9620us) server_negotiation.cc:206] Beginning negotiation
0928 05:20:01.737218 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:20:01.739336 (+ 2118us) server_negotiation.cc:407] Connection header received
0928 05:20:01.741266 (+ 1930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:20:01.741273 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:20:01.741385 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:20:01.741479 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:20:01.743782 (+ 2303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:01.745395 (+ 1613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:01.747747 (+ 2352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:01.748404 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:01.748524 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:01.752918 (+ 4394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:20:01.752925 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:20:01.752928 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:20:01.752958 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:20:01.755549 (+ 2591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:01.756193 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:01.756198 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:01.756202 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:01.756323 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:01.756702 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:01.756707 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:01.756711 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:01.757049 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:20:01.757184 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 05:20:01.757854 (+ 670us) server_negotiation.cc:299] Negotiation successful
0928 05:20:01.758067 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9480,"thread_start_us":11218,"threads_started":1}
I20260928 05:20:01.759415 12938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:01.728195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37687 (local address 127.0.97.1:58769)
0928 05:20:01.738373 (+ 10178us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:01.738430 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 05:20:01.740921 (+ 2491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:20:01.742100 (+ 1179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:20:01.742113 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:20:01.742747 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:20:01.743583 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:20:01.743623 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:20:01.745579 (+ 1956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:20:01.745584 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:20:01.747574 (+ 1990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:20:01.747588 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:20:01.748819 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:20:01.748827 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:20:01.749189 (+ 362us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:01.750003 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:20:01.750033 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:20:01.752662 (+ 2629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:20:01.755665 (+ 3003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:20:01.755674 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:20:01.755690 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:20:01.756084 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:20:01.756421 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:20:01.756426 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:20:01.756430 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:20:01.756603 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:20:01.757165 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:20:01.757172 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:20:01.757513 (+ 341us) client_negotiation.cc:769] Sending connection context
0928 05:20:01.757755 (+ 242us) client_negotiation.cc:240] Negotiation successful
0928 05:20:01.758226 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9858,"thread_start_us":9645,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260928 05:20:01.761183 12935 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37687
I20260928 05:20:01.761458 12935 heartbeater.cc:464] Registering TS with master...
I20260928 05:20:01.762362 12935 heartbeater.cc:511] Master 127.0.97.62:37687 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260928 05:20:01.765177 12722 ts_manager.cc:194] Registered new tserver with Master: 52b3d46fb965452c9ca57b20aad37214 (127.0.97.1:42651)
I20260928 05:20:01.767249 12722 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:58769
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260928 05:20:01.793804 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:47701/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37687
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 05:20:01 dist-test-slave-whsn krb5kdc[12246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572801, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260928 05:20:02.164266 12941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:20:02.164523 12941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:20:02.164673 12941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:20:02.171900 12941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:20:02.172024 12941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:20:02.172173 12941 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 05:20:02.181870 12941 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:47701/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37687
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:20:02.183108 12941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:20:02.184333 12941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:20:02.198653 12946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:20:02.200129 12947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:20:02.207649 12949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:20:02.208014 12941 server_base.cc:1034] running on GCE node
I20260928 05:20:02.208729 12941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:20:02.209718 12941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:20:02.210958 12941 hybrid_clock.cc:648] HybridClock initialized: now 1790572802210899 us; error 51 us; skew 500 ppm
I20260928 05:20:02.211258 12941 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:20:02 dist-test-slave-whsn krb5kdc[12246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572802, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:20:02.214217 12941 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260928 05:20:02.217809 12941 webserver.cc:460] Webserver started at http://127.0.97.2:41407/ using document root <none> and password file <none>
I20260928 05:20:02.218489 12941 fs_manager.cc:359] Metadata directory not provided
I20260928 05:20:02.218601 12941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:20:02.218897 12941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 05:20:02 dist-test-slave-whsn krb5kdc[12246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572802, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260928 05:20:02.240974 12941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ts-1/data/instance:
uuid: "8b90b255839a41178bd941a6b056134d"
format_stamp: "Formatted at 2026-09-28 05:20:02 on dist-test-slave-whsn"
server_key: "b090f7050900aeb2630dbf9278572a73"
server_key_iv: "5346a68ad241615815ec8d2f646bfdbb"
server_key_version: "kuduclusterkey@0"
I20260928 05:20:02.241976 12941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ts-1/wal/instance:
uuid: "8b90b255839a41178bd941a6b056134d"
format_stamp: "Formatted at 2026-09-28 05:20:02 on dist-test-slave-whsn"
server_key: "b090f7050900aeb2630dbf9278572a73"
server_key_iv: "5346a68ad241615815ec8d2f646bfdbb"
server_key_version: "kuduclusterkey@0"
I20260928 05:20:02.249068 12941 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.005s
I20260928 05:20:02.298408 12956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:20:02.300509 12941 fs_manager.cc:725] Time spent opening block manager: real 0.026s user 0.005s sys 0.011s
I20260928 05:20:02.300784 12941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ts-1/wal
uuid: "8b90b255839a41178bd941a6b056134d"
format_stamp: "Formatted at 2026-09-28 05:20:02 on dist-test-slave-whsn"
server_key: "b090f7050900aeb2630dbf9278572a73"
server_key_iv: "5346a68ad241615815ec8d2f646bfdbb"
server_key_version: "kuduclusterkey@0"
I20260928 05:20:02.301227 12941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Sep 28 05:20:02 dist-test-slave-whsn krb5kdc[12246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572801, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260928 05:20:02.362627 12941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:20:02.368364 12941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:20:02.368745 12941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:20:02.407873 12941 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:20:02.428277 12941 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:20:02.428433 12941 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260928 05:20:02.428587 12941 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:20:02.428666 12941 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:20:02.771034 12935 heartbeater.cc:503] Master 127.0.97.62:37687 was elected leader, sending a full tablet report...
I20260928 05:20:02.912604 12941 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35721
I20260928 05:20:02.912657 13077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35721 every 8 connection(s)
I20260928 05:20:02.914278 12941 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ts-1/data/info.pb
I20260928 05:20:02.916383 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 12941
I20260928 05:20:02.916538 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ts-1/wal/instance
I20260928 05:20:02.917251 388 external_mini_cluster.cc:1428] Setting key 9abadd2f232a8498492795b8527d0059
Sep 28 05:20:02 dist-test-slave-whsn krb5kdc[12246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572802, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260928 05:20:02.960872 13081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:02.929085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43087 (local address 127.0.97.62:37687)
0928 05:20:02.933248 (+ 4163us) server_negotiation.cc:206] Beginning negotiation
0928 05:20:02.933255 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:20:02.941895 (+ 8640us) server_negotiation.cc:407] Connection header received
0928 05:20:02.941969 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:20:02.941974 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:20:02.942048 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:20:02.942139 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:20:02.945212 (+ 3073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:02.946342 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:02.948676 (+ 2334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:02.949328 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:02.949424 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:02.953845 (+ 4421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:20:02.953851 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:20:02.953855 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:20:02.953894 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:20:02.957353 (+ 3459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:02.958202 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:02.958208 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:02.958211 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:02.958364 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:02.958868 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:02.958874 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:02.958877 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:02.959193 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:20:02.959311 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 05:20:02.960179 (+ 868us) server_negotiation.cc:299] Negotiation successful
0928 05:20:02.960422 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4011,"thread_start_us":3882,"threads_started":1}
I20260928 05:20:02.963152 13082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:02.929662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37687 (local address 127.0.97.2:43087)
0928 05:20:02.939060 (+ 9398us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:02.939110 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 05:20:02.941549 (+ 2439us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:20:02.942548 (+ 999us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:20:02.942560 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:20:02.943254 (+ 694us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:20:02.944132 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:20:02.944157 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:20:02.946577 (+ 2420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:20:02.946582 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:20:02.948465 (+ 1883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:20:02.948480 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:20:02.949751 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:20:02.949758 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:20:02.950026 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:02.950832 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:20:02.950862 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:20:02.953615 (+ 2753us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:20:02.957493 (+ 3878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:20:02.957501 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:20:02.957518 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:20:02.958053 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:20:02.958503 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:20:02.958507 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:20:02.958511 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:20:02.958750 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:20:02.959401 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:20:02.959409 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:20:02.959784 (+ 375us) client_negotiation.cc:769] Sending connection context
0928 05:20:02.961370 (+ 1586us) client_negotiation.cc:240] Negotiation successful
0928 05:20:02.961865 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9081,"thread_start_us":8851,"threads_started":1}
I20260928 05:20:02.964924 13078 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37687
I20260928 05:20:02.965155 13078 heartbeater.cc:464] Registering TS with master...
I20260928 05:20:02.965931 13078 heartbeater.cc:511] Master 127.0.97.62:37687 requested a full tablet report, sending...
I20260928 05:20:02.967526 12722 ts_manager.cc:194] Registered new tserver with Master: 8b90b255839a41178bd941a6b056134d (127.0.97.2:35721)
I20260928 05:20:02.968528 12722 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:43087
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260928 05:20:02.990509 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:47701/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37687
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:20:03.316238 13084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:20:03.316483 13084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:20:03.316610 13084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:20:03.323400 13084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:20:03.323490 13084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:20:03.323643 13084 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 05:20:03.332623 13084 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:47701/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37687
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:20:03.333777 13084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:20:03.334906 13084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:20:03.346422 13089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:20:03.346467 13090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:20:03.346499 13092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:20:03.347265 13084 server_base.cc:1034] running on GCE node
I20260928 05:20:03.347752 13084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:20:03.348536 13084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:20:03.349718 13084 hybrid_clock.cc:648] HybridClock initialized: now 1790572803349667 us; error 35 us; skew 500 ppm
I20260928 05:20:03.349975 13084 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:20:03 dist-test-slave-whsn krb5kdc[12246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572803, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:20:03.352818 13084 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260928 05:20:03.354126 13084 webserver.cc:460] Webserver started at http://127.0.97.3:36767/ using document root <none> and password file <none>
I20260928 05:20:03.354739 13084 fs_manager.cc:359] Metadata directory not provided
I20260928 05:20:03.354840 13084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:20:03.355104 13084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 05:20:03 dist-test-slave-whsn krb5kdc[12246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572803, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260928 05:20:03.370497 13084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ts-2/data/instance:
uuid: "664827f758474b1da0560894c9c595c5"
format_stamp: "Formatted at 2026-09-28 05:20:03 on dist-test-slave-whsn"
server_key: "f2b55d190e8fe9f73dce0c58e0846731"
server_key_iv: "ce52a24b3f12b56a5bb9be09e96748bc"
server_key_version: "kuduclusterkey@0"
I20260928 05:20:03.371356 13084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ts-2/wal/instance:
uuid: "664827f758474b1da0560894c9c595c5"
format_stamp: "Formatted at 2026-09-28 05:20:03 on dist-test-slave-whsn"
server_key: "f2b55d190e8fe9f73dce0c58e0846731"
server_key_iv: "ce52a24b3f12b56a5bb9be09e96748bc"
server_key_version: "kuduclusterkey@0"
I20260928 05:20:03.376905 13084 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260928 05:20:03.391628 13099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:20:03.393431 13084 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 05:20:03.393615 13084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ts-2/wal
uuid: "664827f758474b1da0560894c9c595c5"
format_stamp: "Formatted at 2026-09-28 05:20:03 on dist-test-slave-whsn"
server_key: "f2b55d190e8fe9f73dce0c58e0846731"
server_key_iv: "ce52a24b3f12b56a5bb9be09e96748bc"
server_key_version: "kuduclusterkey@0"
I20260928 05:20:03.393942 13084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:20:03.413594 13084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:20:03.416750 13084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:20:03.417022 13084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:20:03.417829 13084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:20:03.419028 13084 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:20:03.419106 13084 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:20:03.419224 13084 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:20:03.419274 13084 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:20:03.446790 13084 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39531
I20260928 05:20:03.446832 13211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39531 every 8 connection(s)
I20260928 05:20:03.447974 13084 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ts-2/data/info.pb
I20260928 05:20:03.454903 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 13084
I20260928 05:20:03.455089 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ts-2/wal/instance
I20260928 05:20:03.455596 388 external_mini_cluster.cc:1428] Setting key d89f773324a5c3dd17e42672caae4d1b
Sep 28 05:20:03 dist-test-slave-whsn krb5kdc[12246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572803, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:20:03.466039 13081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:03.450225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34461 (local address 127.0.97.62:37687)
0928 05:20:03.450389 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 05:20:03.450393 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:20:03.451531 (+ 1138us) server_negotiation.cc:407] Connection header received
0928 05:20:03.453225 (+ 1694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:20:03.453229 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:20:03.453296 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:20:03.453383 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:20:03.454709 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:03.455793 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:03.457718 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:03.458381 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:03.458474 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:03.461363 (+ 2889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:20:03.461368 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:20:03.461371 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:20:03.461404 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:20:03.463746 (+ 2342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:03.464299 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:03.464303 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:03.464305 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:03.464415 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:03.464731 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:03.464733 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:03.464735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:03.465011 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:20:03.465112 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 05:20:03.465639 (+ 527us) server_negotiation.cc:299] Negotiation successful
0928 05:20:03.465802 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 05:20:03.466868 13214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:03.450646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37687 (local address 127.0.97.3:34461)
0928 05:20:03.451389 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:03.451425 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 05:20:03.453002 (+ 1577us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:20:03.453552 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:20:03.453560 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:20:03.453988 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:20:03.454561 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:20:03.454579 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:20:03.455971 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:20:03.455975 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:20:03.457587 (+ 1612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:20:03.457602 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:20:03.458564 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:20:03.458569 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:20:03.458768 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:03.459316 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:20:03.459336 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:20:03.461183 (+ 1847us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:20:03.463848 (+ 2665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:20:03.463854 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:20:03.463864 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:20:03.464216 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:20:03.464486 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:20:03.464489 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:20:03.464491 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:20:03.464655 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:20:03.465105 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:20:03.465110 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:20:03.465368 (+ 258us) client_negotiation.cc:769] Sending connection context
0928 05:20:03.465558 (+ 190us) client_negotiation.cc:240] Negotiation successful
0928 05:20:03.465878 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":272,"threads_started":1}
I20260928 05:20:03.467995 13212 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37687
I20260928 05:20:03.468245 13212 heartbeater.cc:464] Registering TS with master...
I20260928 05:20:03.468942 13212 heartbeater.cc:511] Master 127.0.97.62:37687 requested a full tablet report, sending...
I20260928 05:20:03.470292 12721 ts_manager.cc:194] Registered new tserver with Master: 664827f758474b1da0560894c9c595c5 (127.0.97.3:39531)
I20260928 05:20:03.471148 12721 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:34461
I20260928 05:20:03.481997 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 05:20:03.483539 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 12802
Sep 28 05:20:03 dist-test-slave-whsn krb5kdc[12246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572799, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260928 05:20:03.497260 13216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:03.484279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35780 (local address 127.0.97.1:42651)
0928 05:20:03.484762 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0928 05:20:03.484768 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:20:03.484790 (+ 22us) server_negotiation.cc:407] Connection header received
0928 05:20:03.484905 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:20:03.484913 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:20:03.485081 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:20:03.485196 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:20:03.486470 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:03.487417 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:03.488671 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:03.489198 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:03.489287 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:03.492768 (+ 3481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:20:03.492778 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:20:03.492789 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:20:03.492820 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:20:03.495174 (+ 2354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:03.495710 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:03.495715 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:03.495718 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:03.495831 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:03.496085 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:03.496087 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:03.496089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:03.496360 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:20:03.496568 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0928 05:20:03.496729 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 05:20:03.496937 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":228,"threads_started":1}
I20260928 05:20:03.498327 12930 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:20:03.577570 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 12802
I20260928 05:20:03.589457 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 12941
Sep 28 05:20:03 dist-test-slave-whsn krb5kdc[12246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572799, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260928 05:20:03.604447 13218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:03.590266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45896 (local address 127.0.97.2:35721)
0928 05:20:03.590805 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0928 05:20:03.590812 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:20:03.590835 (+ 23us) server_negotiation.cc:407] Connection header received
0928 05:20:03.590891 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:20:03.590897 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:20:03.591101 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:20:03.591216 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:20:03.592316 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:03.593353 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:03.594782 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:03.595368 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:03.595481 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:03.599367 (+ 3886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:20:03.599377 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:20:03.599390 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:20:03.599419 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:20:03.601973 (+ 2554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:03.602568 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:03.602575 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:03.602577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:03.602697 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:03.603012 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:03.603016 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:03.603018 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:03.603330 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:20:03.603595 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0928 05:20:03.603887 (+ 292us) server_negotiation.cc:299] Negotiation successful
0928 05:20:03.604130 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":271,"threads_started":1}
I20260928 05:20:03.605783 13073 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:20:03.683959 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 12941
I20260928 05:20:03.695539 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 13084
Sep 28 05:20:03 dist-test-slave-whsn krb5kdc[12246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572799, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260928 05:20:03.709664 13220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:03.696433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37774 (local address 127.0.97.3:39531)
0928 05:20:03.697086 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0928 05:20:03.697093 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:20:03.697120 (+ 27us) server_negotiation.cc:407] Connection header received
0928 05:20:03.697174 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:20:03.697179 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:20:03.697381 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:20:03.697494 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:20:03.698739 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:03.699687 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:03.700835 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:03.701400 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:03.701491 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:03.705252 (+ 3761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:20:03.705261 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:20:03.705274 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:20:03.705304 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:20:03.707609 (+ 2305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:03.708137 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:03.708143 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:03.708145 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:03.708254 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:03.708522 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:03.708525 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:03.708526 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:03.708808 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:20:03.709074 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0928 05:20:03.709226 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 05:20:03.709397 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":345,"threads_started":1}
I20260928 05:20:03.710834 13207 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:20:03.794100 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 13084
I20260928 05:20:03.806915 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 12690
I20260928 05:20:03.808080 12752 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:20:03.910509 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 12690
I20260928 05:20:03.928287 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37687
--webserver_interface=127.0.97.62
--webserver_port=46749
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37687
--ranger_config_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:47701/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:47701/kms with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:20:04.224917 13223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:20:04.225184 13223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:20:04.225315 13223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:20:04.232205 13223 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:20:04.232302 13223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:20:04.232372 13223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:20:04.232445 13223 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:20:04.232503 13223 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:20:04.241853 13223 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:47701/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37687
--ranger_config_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=46749
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:20:04.242959 13223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:20:04.244199 13223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:20:04.254765 13228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:20:04.254845 13229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:20:04.255059 13231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:20:04.255714 13223 server_base.cc:1034] running on GCE node
I20260928 05:20:04.256378 13223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:20:04.257680 13223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:20:04.258862 13223 hybrid_clock.cc:648] HybridClock initialized: now 1790572804258818 us; error 40 us; skew 500 ppm
I20260928 05:20:04.259147 13223 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:20:04 dist-test-slave-whsn krb5kdc[12246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572804, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:20:04.261930 13223 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 05:20:04.263186 13223 webserver.cc:460] Webserver started at http://127.0.97.62:46749/ using document root <none> and password file <none>
I20260928 05:20:04.263765 13223 fs_manager.cc:359] Metadata directory not provided
I20260928 05:20:04.263880 13223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 05:20:04 dist-test-slave-whsn krb5kdc[12246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572804, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260928 05:20:04.325448 13223 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 05:20:04.328280 13239 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:20:04.329581 13223 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:20:04.329744 13223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/master-0/wal
uuid: "30e4edfa8186490b8cb4258d9524eae7"
format_stamp: "Formatted at 2026-09-28 05:20:00 on dist-test-slave-whsn"
server_key: "f8f9f02bebbb1cf53fdf44903a886d96"
server_key_iv: "fd0a719e426dad0385eef8dcb46637d6"
server_key_version: "kuduclusterkey@0"
I20260928 05:20:04.330643 13223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:20:04.366282 13223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:20:04.369694 13223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:20:04.369946 13223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:20:04.386112 13223 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37687
I20260928 05:20:04.386157 13290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37687 every 8 connection(s)
I20260928 05:20:04.387455 13223 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:20:04.393935 13291 sys_catalog.cc:263] Verifying existing consensus state
I20260928 05:20:04.396714 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 13223
I20260928 05:20:04.397100 13291 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30e4edfa8186490b8cb4258d9524eae7: Bootstrap starting.
I20260928 05:20:04.397531 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:47701/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:42651
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:37687
--webserver_port=41437
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:47701/kms with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 05:20:04.414587 13291 log.cc:824] T 00000000000000000000000000000000 P 30e4edfa8186490b8cb4258d9524eae7: Log is configured to *not* fsync() on all Append() calls
I20260928 05:20:04.422721 13291 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30e4edfa8186490b8cb4258d9524eae7: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 05:20:04.423182 13291 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30e4edfa8186490b8cb4258d9524eae7: Bootstrap complete.
I20260928 05:20:04.431711 13291 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30e4edfa8186490b8cb4258d9524eae7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30e4edfa8186490b8cb4258d9524eae7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37687 } }
I20260928 05:20:04.432495 13291 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30e4edfa8186490b8cb4258d9524eae7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30e4edfa8186490b8cb4258d9524eae7, State: Initialized, Role: FOLLOWER
I20260928 05:20:04.433097 13291 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30e4edfa8186490b8cb4258d9524eae7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30e4edfa8186490b8cb4258d9524eae7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37687 } }
I20260928 05:20:04.433331 13291 raft_consensus.cc:388] T 00000000000000000000000000000000 P 30e4edfa8186490b8cb4258d9524eae7 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:20:04.433426 13291 raft_consensus.cc:482] T 00000000000000000000000000000000 P 30e4edfa8186490b8cb4258d9524eae7 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:20:04.433583 13291 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30e4edfa8186490b8cb4258d9524eae7 [term 1 FOLLOWER]: Advancing to term 2
I20260928 05:20:04.437219 13291 raft_consensus.cc:504] T 00000000000000000000000000000000 P 30e4edfa8186490b8cb4258d9524eae7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30e4edfa8186490b8cb4258d9524eae7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37687 } }
I20260928 05:20:04.437680 13291 leader_election.cc:302] T 00000000000000000000000000000000 P 30e4edfa8186490b8cb4258d9524eae7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30e4edfa8186490b8cb4258d9524eae7; no voters:
I20260928 05:20:04.438241 13291 leader_election.cc:288] T 00000000000000000000000000000000 P 30e4edfa8186490b8cb4258d9524eae7 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 05:20:04.438429 13295 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 30e4edfa8186490b8cb4258d9524eae7 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 05:20:04.439343 13295 raft_consensus.cc:686] T 00000000000000000000000000000000 P 30e4edfa8186490b8cb4258d9524eae7 [term 2 LEADER]: Becoming Leader. State: Replica: 30e4edfa8186490b8cb4258d9524eae7, State: Running, Role: LEADER
I20260928 05:20:04.440111 13291 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30e4edfa8186490b8cb4258d9524eae7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:20:04.439971 13295 consensus_queue.cc:237] T 00000000000000000000000000000000 P 30e4edfa8186490b8cb4258d9524eae7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30e4edfa8186490b8cb4258d9524eae7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37687 } }
I20260928 05:20:04.442490 13297 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30e4edfa8186490b8cb4258d9524eae7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30e4edfa8186490b8cb4258d9524eae7. Latest consensus state: current_term: 2 leader_uuid: "30e4edfa8186490b8cb4258d9524eae7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30e4edfa8186490b8cb4258d9524eae7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37687 } } }
I20260928 05:20:04.442919 13297 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30e4edfa8186490b8cb4258d9524eae7 [sys.catalog]: This master's current role is: LEADER
I20260928 05:20:04.443473 13296 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30e4edfa8186490b8cb4258d9524eae7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "30e4edfa8186490b8cb4258d9524eae7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30e4edfa8186490b8cb4258d9524eae7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37687 } } }
I20260928 05:20:04.443790 13296 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30e4edfa8186490b8cb4258d9524eae7 [sys.catalog]: This master's current role is: LEADER
I20260928 05:20:04.450842 13301 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:20:04.456915 13301 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:20:04.458329 13301 catalog_manager.cc:1259] Loaded cluster ID: 6eedfb183f36483d9878dc64bc8b5bf9
I20260928 05:20:04.458483 13301 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:20:04.462536 13301 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:20:04.464038 13301 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 30e4edfa8186490b8cb4258d9524eae7: Loaded TSK: 0
I20260928 05:20:04.465663 13301 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 05:20:04.470599 13291 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260928 05:20:04.845464 13293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:20:04.845918 13293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:20:04.846143 13293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:20:04.857312 13293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:20:04.857517 13293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:20:04.857754 13293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 05:20:04.872998 13293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:47701/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:42651
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=41437
--tserver_master_addrs=127.0.97.62:37687
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:20:04.874852 13293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:20:04.876598 13293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:20:04.896201 13329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:20:04.897501 13330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:20:04.898401 13333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:20:04.898612 13293 server_base.cc:1034] running on GCE node
I20260928 05:20:04.899223 13293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:20:04.900156 13293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:20:04.901351 13293 hybrid_clock.cc:648] HybridClock initialized: now 1790572804901284 us; error 61 us; skew 500 ppm
I20260928 05:20:04.901645 13293 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:20:04 dist-test-slave-whsn krb5kdc[12246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572804, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:20:04.904593 13293 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260928 05:20:04.906152 13293 webserver.cc:460] Webserver started at http://127.0.97.1:41437/ using document root <none> and password file <none>
I20260928 05:20:04.906808 13293 fs_manager.cc:359] Metadata directory not provided
I20260928 05:20:04.906927 13293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 05:20:04 dist-test-slave-whsn krb5kdc[12246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572804, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260928 05:20:04.940685 13293 fs_manager.cc:709] Time spent opening directory manager: real 0.017s user 0.003s sys 0.012s
I20260928 05:20:04.948323 13340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:20:04.949954 13293 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.006s sys 0.001s
I20260928 05:20:04.950214 13293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ts-0/wal
uuid: "52b3d46fb965452c9ca57b20aad37214"
format_stamp: "Formatted at 2026-09-28 05:20:01 on dist-test-slave-whsn"
server_key: "4968edef52267842c187378b2f5bccaf"
server_key_iv: "a9879d92b042eb9570b901041b1feebe"
server_key_version: "kuduclusterkey@0"
I20260928 05:20:04.951337 13293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:20:04.976218 13293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:20:04.981336 13293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:20:04.981681 13293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:20:04.984210 13293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:20:04.986663 13293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:20:04.986814 13293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260928 05:20:04.986946 13293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:20:04.987046 13293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:20:05.056483 13293 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42651
I20260928 05:20:05.056537 13453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42651 every 8 connection(s)
I20260928 05:20:05.058012 13293 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ts-0/data/info.pb
I20260928 05:20:05.066022 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 13293
I20260928 05:20:05.066811 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:47701/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:35721
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:37687
--webserver_port=41407
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:47701/kms with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 05:20:05 dist-test-slave-whsn krb5kdc[12246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572804, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:20:05.087152 13456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:05.061545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37687 (local address 127.0.97.1:49565)
0928 05:20:05.062406 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:05.062453 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 05:20:05.064748 (+ 2295us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:20:05.067402 (+ 2654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:20:05.067414 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:20:05.067998 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:20:05.068898 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:20:05.068923 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:20:05.071213 (+ 2290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:20:05.071219 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:20:05.072804 (+ 1585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:20:05.072816 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:20:05.074276 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:20:05.074297 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:20:05.074573 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:05.075360 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:20:05.075389 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:20:05.077897 (+ 2508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:20:05.082168 (+ 4271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:20:05.082175 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:20:05.082192 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:20:05.082775 (+ 583us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:20:05.083269 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:20:05.083275 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:20:05.083279 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:20:05.083503 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:20:05.084460 (+ 957us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:20:05.084468 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:20:05.084890 (+ 422us) client_negotiation.cc:769] Sending connection context
0928 05:20:05.085191 (+ 301us) client_negotiation.cc:240] Negotiation successful
0928 05:20:05.085768 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":324,"threads_started":1}
I20260928 05:20:05.087141 13457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:05.063015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49565 (local address 127.0.97.62:37687)
0928 05:20:05.063713 (+ 698us) server_negotiation.cc:206] Beginning negotiation
0928 05:20:05.063730 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 05:20:05.063781 (+ 51us) server_negotiation.cc:407] Connection header received
0928 05:20:05.065521 (+ 1740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:20:05.065540 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:20:05.066005 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:20:05.066466 (+ 461us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:20:05.069124 (+ 2658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:05.071027 (+ 1903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:05.072946 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:05.073987 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:05.074175 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:05.078197 (+ 4022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:20:05.078208 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:20:05.078228 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:20:05.078283 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:20:05.082003 (+ 3720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:05.082941 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:05.082949 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:05.082956 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:05.083144 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:05.083621 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:05.083625 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:05.083628 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:05.084221 (+ 593us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:20:05.084389 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0928 05:20:05.085443 (+ 1054us) server_negotiation.cc:299] Negotiation successful
0928 05:20:05.085840 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":253,"threads_started":1}
I20260928 05:20:05.090776 13454 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37687
I20260928 05:20:05.091104 13454 heartbeater.cc:464] Registering TS with master...
I20260928 05:20:05.091982 13454 heartbeater.cc:511] Master 127.0.97.62:37687 requested a full tablet report, sending...
W20260928 05:20:05.095260 13454 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:37687 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 05:20:05.095474 13454 heartbeater.cc:464] Registering TS with master...
I20260928 05:20:05.095727 13454 heartbeater.cc:511] Master 127.0.97.62:37687 requested a full tablet report, sending...
I20260928 05:20:05.097059 13454 heartbeater.cc:464] Registering TS with master...
I20260928 05:20:05.097399 13454 heartbeater.cc:511] Master 127.0.97.62:37687 requested a full tablet report, sending...
W20260928 05:20:05.098464 13454 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 05:20:05.436534 13458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:20:05.436772 13458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:20:05.436909 13458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:20:05.444622 13458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:20:05.444730 13458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:20:05.444913 13458 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 05:20:05.454591 13458 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:47701/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:35721
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=41407
--tserver_master_addrs=127.0.97.62:37687
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:20:05.455928 13458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:20:05.457378 13458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:20:05.473382 13470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:20:05.473857 13471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:20:05.474308 13474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:20:05.475346 13458 server_base.cc:1034] running on GCE node
I20260928 05:20:05.476037 13458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:20:05.477206 13458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:20:05.478442 13458 hybrid_clock.cc:648] HybridClock initialized: now 1790572805478384 us; error 43 us; skew 500 ppm
I20260928 05:20:05.478796 13458 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:20:05 dist-test-slave-whsn krb5kdc[12246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572805, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:20:05.487317 13458 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260928 05:20:05.489115 13458 webserver.cc:460] Webserver started at http://127.0.97.2:41407/ using document root <none> and password file <none>
I20260928 05:20:05.489959 13458 fs_manager.cc:359] Metadata directory not provided
I20260928 05:20:05.490110 13458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 05:20:05 dist-test-slave-whsn krb5kdc[12246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572805, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260928 05:20:05.517437 13458 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.000s sys 0.004s
I20260928 05:20:05.520072 13483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:20:05.521417 13458 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:20:05.521559 13458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ts-1/wal
uuid: "8b90b255839a41178bd941a6b056134d"
format_stamp: "Formatted at 2026-09-28 05:20:02 on dist-test-slave-whsn"
server_key: "b090f7050900aeb2630dbf9278572a73"
server_key_iv: "5346a68ad241615815ec8d2f646bfdbb"
server_key_version: "kuduclusterkey@0"
I20260928 05:20:05.522466 13458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:20:05.547055 13458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:20:05.551726 13458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:20:05.552001 13458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:20:05.560957 13458 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:20:05.563294 13458 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:20:05.563380 13458 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:20:05.563480 13458 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:20:05.563548 13458 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:20:05.606168 13458 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35721
I20260928 05:20:05.606217 13597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35721 every 8 connection(s)
I20260928 05:20:05.607930 13458 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ts-1/data/info.pb
I20260928 05:20:05.617429 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 13458
I20260928 05:20:05.618211 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:47701/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:39531
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:37687
--webserver_port=36767
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:47701/kms with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 05:20:05 dist-test-slave-whsn krb5kdc[12246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572805, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:20:05.646054 13600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:05.610704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38375 (local address 127.0.97.62:37687)
0928 05:20:05.623051 (+ 12347us) server_negotiation.cc:206] Beginning negotiation
0928 05:20:05.623058 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:20:05.623822 (+ 764us) server_negotiation.cc:407] Connection header received
0928 05:20:05.626440 (+ 2618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:20:05.626446 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:20:05.626536 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:20:05.626622 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:20:05.630316 (+ 3694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:05.631474 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:05.633919 (+ 2445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:05.634566 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:05.634662 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:05.638969 (+ 4307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:20:05.638977 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:20:05.638982 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:20:05.639020 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:20:05.642048 (+ 3028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:05.643221 (+ 1173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:05.643227 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:05.643231 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:05.643404 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:05.643946 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:05.643951 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:05.643955 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:05.644290 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:20:05.644436 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 05:20:05.645413 (+ 977us) server_negotiation.cc:299] Negotiation successful
0928 05:20:05.645629 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12204,"thread_start_us":12339,"threads_started":1}
I20260928 05:20:05.648408 13601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:05.611302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37687 (local address 127.0.97.2:38375)
0928 05:20:05.623635 (+ 12333us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:05.623689 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 05:20:05.626112 (+ 2423us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:20:05.627171 (+ 1059us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:20:05.627183 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:20:05.627745 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:20:05.628529 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:20:05.628555 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:20:05.631684 (+ 3129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:20:05.631690 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:20:05.633742 (+ 2052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:20:05.633768 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:20:05.634986 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:20:05.634992 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:20:05.635277 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:05.636046 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:20:05.636075 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:20:05.638714 (+ 2639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:20:05.642485 (+ 3771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:20:05.642494 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:20:05.642510 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:20:05.643082 (+ 572us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:20:05.643554 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:20:05.643559 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:20:05.643563 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:20:05.643809 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:20:05.644528 (+ 719us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:20:05.644536 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:20:05.644985 (+ 449us) client_negotiation.cc:769] Sending connection context
0928 05:20:05.646490 (+ 1505us) client_negotiation.cc:240] Negotiation successful
0928 05:20:05.647053 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12050,"thread_start_us":11864,"threads_started":1}
I20260928 05:20:05.650157 13598 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37687
I20260928 05:20:05.650475 13598 heartbeater.cc:464] Registering TS with master...
I20260928 05:20:05.651319 13598 heartbeater.cc:511] Master 127.0.97.62:37687 requested a full tablet report, sending...
I20260928 05:20:05.653380 13256 ts_manager.cc:194] Registered new tserver with Master: 8b90b255839a41178bd941a6b056134d (127.0.97.2:35721)
I20260928 05:20:05.655491 13256 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:38375
Sep 28 05:20:05 dist-test-slave-whsn krb5kdc[12246](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790572805, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260928 05:20:05.997537 13602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:20:05.997798 13602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:20:05.997928 13602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:20:06.005630 13602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:20:06.005743 13602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:20:06.005892 13602 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 05:20:06.017127 13602 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:47701/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:39531
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=36767
--tserver_master_addrs=127.0.97.62:37687
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:20:06.018687 13602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:20:06.020079 13602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:20:06.035133 13608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:20:06.038093 13609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:20:06.048709 13602 server_base.cc:1034] running on GCE node
W20260928 05:20:06.049114 13611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:20:06.049896 13602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:20:06.051182 13602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:20:06.052433 13602 hybrid_clock.cc:648] HybridClock initialized: now 1790572806052372 us; error 49 us; skew 500 ppm
I20260928 05:20:06.052846 13602 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:20:06 dist-test-slave-whsn krb5kdc[12246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572806, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:20:06.056923 13602 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260928 05:20:06.069224 13602 webserver.cc:460] Webserver started at http://127.0.97.3:36767/ using document root <none> and password file <none>
I20260928 05:20:06.070150 13602 fs_manager.cc:359] Metadata directory not provided
I20260928 05:20:06.070287 13602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 05:20:06 dist-test-slave-whsn krb5kdc[12246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572806, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260928 05:20:06.097790 13602 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.002s sys 0.001s
I20260928 05:20:06.098606 13454 heartbeater.cc:464] Registering TS with master...
I20260928 05:20:06.099007 13454 heartbeater.cc:511] Master 127.0.97.62:37687 requested a full tablet report, sending...
I20260928 05:20:06.101234 13256 ts_manager.cc:194] Registered new tserver with Master: 52b3d46fb965452c9ca57b20aad37214 (127.0.97.1:42651)
Sep 28 05:20:06 dist-test-slave-whsn krb5kdc[12246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572805, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260928 05:20:06.102363 13256 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:49565
I20260928 05:20:06.105705 13619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:20:06.107498 13602 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.006s sys 0.000s
I20260928 05:20:06.107724 13602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ts-2/wal
uuid: "664827f758474b1da0560894c9c595c5"
format_stamp: "Formatted at 2026-09-28 05:20:03 on dist-test-slave-whsn"
server_key: "f2b55d190e8fe9f73dce0c58e0846731"
server_key_iv: "ce52a24b3f12b56a5bb9be09e96748bc"
server_key_version: "kuduclusterkey@0"
I20260928 05:20:06.109089 13602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:20:06.149592 13602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:20:06.154439 13602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:20:06.154745 13602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:20:06.162053 13602 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:20:06.167014 13602 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:20:06.167125 13602 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:20:06.167240 13602 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:20:06.167320 13602 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:20:06.341922 13602 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39531
I20260928 05:20:06.341971 13731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39531 every 8 connection(s)
I20260928 05:20:06.343611 13602 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ts-2/data/info.pb
I20260928 05:20:06.345069 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 13602
Sep 28 05:20:06 dist-test-slave-whsn krb5kdc[12246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572806, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:20:06.383370 13734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:06.346580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40276 (local address 127.0.97.62:37687)
0928 05:20:06.366729 (+ 20149us) server_negotiation.cc:206] Beginning negotiation
0928 05:20:06.366735 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:20:06.366764 (+ 29us) server_negotiation.cc:407] Connection header received
0928 05:20:06.366838 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:20:06.366843 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:20:06.366914 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:20:06.367478 (+ 564us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:20:06.369518 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:06.370545 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:06.372521 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:06.373175 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:06.373297 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:06.376807 (+ 3510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:20:06.376813 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:20:06.376817 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:20:06.376869 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:20:06.380123 (+ 3254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:06.380822 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:06.380828 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:06.380832 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:06.380990 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:06.381821 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:06.381826 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:06.381830 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:06.382189 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:20:06.382452 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0928 05:20:06.382689 (+ 237us) server_negotiation.cc:299] Negotiation successful
0928 05:20:06.382946 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20034,"thread_start_us":20435,"threads_started":1}
I20260928 05:20:06.388592 13737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:06.364998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42359 (local address 127.0.97.62:37687)
0928 05:20:06.369475 (+ 4477us) server_negotiation.cc:206] Beginning negotiation
0928 05:20:06.369481 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:20:06.370209 (+ 728us) server_negotiation.cc:407] Connection header received
0928 05:20:06.372603 (+ 2394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:20:06.372608 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:20:06.372684 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:20:06.372775 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:20:06.374595 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:06.375707 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:06.377672 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:06.378146 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:06.378261 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:06.381621 (+ 3360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:20:06.381627 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:20:06.381632 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:20:06.381666 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:20:06.385528 (+ 3862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:06.386429 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:06.386434 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:06.386437 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:06.386572 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:06.386989 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:06.386995 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:06.386998 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:06.387257 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:20:06.387396 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 05:20:06.388034 (+ 638us) server_negotiation.cc:299] Negotiation successful
0928 05:20:06.388225 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4350,"thread_start_us":4223,"threads_started":1}
I20260928 05:20:06.390040 13736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:06.364170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37687 (local address 127.0.97.3:42359)
0928 05:20:06.369990 (+ 5820us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:06.370041 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 05:20:06.372339 (+ 2298us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:20:06.373050 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:20:06.373059 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:20:06.373584 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:20:06.374383 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:20:06.374410 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:20:06.375896 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:20:06.375901 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:20:06.377514 (+ 1613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:20:06.377528 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:20:06.378380 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:20:06.378387 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:20:06.378657 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:06.379208 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:20:06.379233 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:20:06.381366 (+ 2133us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:20:06.385677 (+ 4311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:20:06.385686 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:20:06.385703 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:20:06.386289 (+ 586us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:20:06.386666 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:20:06.386670 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:20:06.386674 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:20:06.386876 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:20:06.387375 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:20:06.387383 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:20:06.387704 (+ 321us) client_negotiation.cc:769] Sending connection context
0928 05:20:06.387930 (+ 226us) client_negotiation.cc:240] Negotiation successful
0928 05:20:06.388442 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5541,"thread_start_us":5294,"threads_started":1}
I20260928 05:20:06.391748 13732 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37687
I20260928 05:20:06.392045 13732 heartbeater.cc:464] Registering TS with master...
I20260928 05:20:06.392804 13732 heartbeater.cc:511] Master 127.0.97.62:37687 requested a full tablet report, sending...
I20260928 05:20:06.394431 13256 ts_manager.cc:194] Registered new tserver with Master: 664827f758474b1da0560894c9c595c5 (127.0.97.3:39531)
I20260928 05:20:06.395460 13256 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:42359
I20260928 05:20:06.400276 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 05:20:06.429488 13737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:06.404225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40292 (local address 127.0.97.62:37687)
0928 05:20:06.404405 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 05:20:06.404410 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:20:06.414220 (+ 9810us) server_negotiation.cc:407] Connection header received
0928 05:20:06.414530 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:20:06.414536 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:20:06.414627 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:20:06.414743 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:20:06.416755 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:06.417802 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:06.419680 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:06.420195 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:06.420271 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:06.422881 (+ 2610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:20:06.422886 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:20:06.422889 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:20:06.422920 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:20:06.425933 (+ 3013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:06.426634 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:06.426640 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:06.426643 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:06.426790 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:06.427207 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:06.427212 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:06.427215 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:06.427576 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:20:06.427808 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0928 05:20:06.428991 (+ 1183us) server_negotiation.cc:299] Negotiation successful
0928 05:20:06.429202 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 05:20:06.434897 13256 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40292:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 05:20:06.442893 13256 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
Sep 28 05:20:06 dist-test-slave-whsn krb5kdc[12246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572805, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260928 05:20:06.545956 13748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:06.527285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35721 (local address 127.0.0.1:45912)
0928 05:20:06.533465 (+ 6180us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:06.533528 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0928 05:20:06.533704 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:20:06.537817 (+ 4113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:20:06.537825 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:20:06.537856 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:20:06.538253 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:20:06.538269 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:20:06.540308 (+ 2039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:20:06.540313 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:20:06.542784 (+ 2471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:20:06.542802 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:20:06.545048 (+ 2246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:20:06.545055 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:20:06.545236 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:06.545292 (+ 56us) client_negotiation.cc:769] Sending connection context
0928 05:20:06.545414 (+ 122us) client_negotiation.cc:240] Negotiation successful
0928 05:20:06.545596 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5868,"thread_start_us":5725,"threads_started":1}
I20260928 05:20:06.546805 13749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:06.527417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45912 (local address 127.0.97.2:35721)
0928 05:20:06.537202 (+ 9785us) server_negotiation.cc:206] Beginning negotiation
0928 05:20:06.537211 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:20:06.537240 (+ 29us) server_negotiation.cc:407] Connection header received
0928 05:20:06.537335 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:20:06.537342 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:20:06.537600 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:20:06.537809 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:20:06.538434 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:06.540161 (+ 1727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:06.543011 (+ 2850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:06.544554 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:06.544671 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:06.544903 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0928 05:20:06.546325 (+ 1422us) server_negotiation.cc:299] Negotiation successful
0928 05:20:06.546495 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9633,"thread_start_us":9527,"threads_started":1}
I20260928 05:20:06.550992 13533 tablet_service.cc:1467] Processing CreateTablet for tablet 3a5acaa9f2a1449f8f44305726089e3f (DEFAULT_TABLE table=test-table [id=e157fea335f14be980b488b03c3e3223]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:20:06.554378 13533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a5acaa9f2a1449f8f44305726089e3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:20:06.566124 13751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:06.529997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37778 (local address 127.0.97.3:39531)
0928 05:20:06.534296 (+ 4299us) server_negotiation.cc:206] Beginning negotiation
0928 05:20:06.534305 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:20:06.544671 (+ 10366us) server_negotiation.cc:407] Connection header received
0928 05:20:06.544841 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:20:06.544888 (+ 47us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:20:06.545159 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:20:06.545379 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:20:06.551742 (+ 6363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:06.553401 (+ 1659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:06.563161 (+ 9760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:06.564650 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:06.564754 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:06.564993 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0928 05:20:06.565597 (+ 604us) server_negotiation.cc:299] Negotiation successful
0928 05:20:06.565763 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4108,"thread_start_us":4470,"threads_started":1}
I20260928 05:20:06.567107 13750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:06.529368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39531 (local address 127.0.0.1:37778)
0928 05:20:06.544558 (+ 15190us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:06.544576 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 05:20:06.544688 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:20:06.550141 (+ 5453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:20:06.550146 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:20:06.550176 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:20:06.550520 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:20:06.550530 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:20:06.560946 (+ 10416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:20:06.560952 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:20:06.562966 (+ 2014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:20:06.562981 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:20:06.565188 (+ 2207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:20:06.565195 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:20:06.565381 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:06.565415 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 05:20:06.566613 (+ 1198us) client_negotiation.cc:240] Negotiation successful
0928 05:20:06.566756 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15076,"thread_start_us":15402,"threads_started":1}
I20260928 05:20:06.572152 13667 tablet_service.cc:1467] Processing CreateTablet for tablet 3a5acaa9f2a1449f8f44305726089e3f (DEFAULT_TABLE table=test-table [id=e157fea335f14be980b488b03c3e3223]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:20:06.573619 13667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a5acaa9f2a1449f8f44305726089e3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:20:06.583124 13753 tablet_bootstrap.cc:492] T 3a5acaa9f2a1449f8f44305726089e3f P 8b90b255839a41178bd941a6b056134d: Bootstrap starting.
I20260928 05:20:06.586020 13748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:06.553416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42651 (local address 127.0.0.1:35786)
0928 05:20:06.553693 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:06.553709 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 05:20:06.553798 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:20:06.563255 (+ 9457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:20:06.563260 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:20:06.563285 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:20:06.563440 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:20:06.563449 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:20:06.565540 (+ 2091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:20:06.565546 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:20:06.567371 (+ 1825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:20:06.567384 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:20:06.585219 (+ 17835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:20:06.585227 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:20:06.585449 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:06.585489 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 05:20:06.585585 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 05:20:06.585743 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180}
I20260928 05:20:06.587030 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:06.545468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35786 (local address 127.0.97.1:42651)
0928 05:20:06.562530 (+ 17062us) server_negotiation.cc:206] Beginning negotiation
0928 05:20:06.562539 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:20:06.562566 (+ 27us) server_negotiation.cc:407] Connection header received
0928 05:20:06.562651 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:20:06.562658 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:20:06.562880 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:20:06.563606 (+ 726us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:20:06.563832 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:06.565402 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:06.580503 (+ 15101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:06.582011 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:06.585197 (+ 3186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:06.585441 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0928 05:20:06.586542 (+ 1101us) server_negotiation.cc:299] Negotiation successful
0928 05:20:06.586712 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16895,"thread_start_us":17536,"threads_started":1}
I20260928 05:20:06.589042 13753 tablet_bootstrap.cc:654] T 3a5acaa9f2a1449f8f44305726089e3f P 8b90b255839a41178bd941a6b056134d: Neither blocks nor log segments found. Creating new log.
I20260928 05:20:06.590387 13753 log.cc:824] T 3a5acaa9f2a1449f8f44305726089e3f P 8b90b255839a41178bd941a6b056134d: Log is configured to *not* fsync() on all Append() calls
I20260928 05:20:06.590775 13389 tablet_service.cc:1467] Processing CreateTablet for tablet 3a5acaa9f2a1449f8f44305726089e3f (DEFAULT_TABLE table=test-table [id=e157fea335f14be980b488b03c3e3223]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:20:06.593469 13389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a5acaa9f2a1449f8f44305726089e3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:20:06.599560 13753 tablet_bootstrap.cc:492] T 3a5acaa9f2a1449f8f44305726089e3f P 8b90b255839a41178bd941a6b056134d: No bootstrap required, opened a new log
I20260928 05:20:06.599839 13753 ts_tablet_manager.cc:1397] T 3a5acaa9f2a1449f8f44305726089e3f P 8b90b255839a41178bd941a6b056134d: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.011s
I20260928 05:20:06.620102 13755 tablet_bootstrap.cc:492] T 3a5acaa9f2a1449f8f44305726089e3f P 664827f758474b1da0560894c9c595c5: Bootstrap starting.
I20260928 05:20:06.623517 13753 raft_consensus.cc:348] T 3a5acaa9f2a1449f8f44305726089e3f P 8b90b255839a41178bd941a6b056134d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b90b255839a41178bd941a6b056134d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35721 } } peers { permanent_uuid: "664827f758474b1da0560894c9c595c5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39531 } } peers { permanent_uuid: "52b3d46fb965452c9ca57b20aad37214" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42651 } }
I20260928 05:20:06.625491 13753 raft_consensus.cc:374] T 3a5acaa9f2a1449f8f44305726089e3f P 8b90b255839a41178bd941a6b056134d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:20:06.625634 13753 raft_consensus.cc:729] T 3a5acaa9f2a1449f8f44305726089e3f P 8b90b255839a41178bd941a6b056134d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b90b255839a41178bd941a6b056134d, State: Initialized, Role: FOLLOWER
I20260928 05:20:06.626242 13755 tablet_bootstrap.cc:654] T 3a5acaa9f2a1449f8f44305726089e3f P 664827f758474b1da0560894c9c595c5: Neither blocks nor log segments found. Creating new log.
I20260928 05:20:06.626361 13753 consensus_queue.cc:260] T 3a5acaa9f2a1449f8f44305726089e3f P 8b90b255839a41178bd941a6b056134d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b90b255839a41178bd941a6b056134d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35721 } } peers { permanent_uuid: "664827f758474b1da0560894c9c595c5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39531 } } peers { permanent_uuid: "52b3d46fb965452c9ca57b20aad37214" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42651 } }
I20260928 05:20:06.628661 13756 tablet_bootstrap.cc:492] T 3a5acaa9f2a1449f8f44305726089e3f P 52b3d46fb965452c9ca57b20aad37214: Bootstrap starting.
I20260928 05:20:06.632171 13756 tablet_bootstrap.cc:654] T 3a5acaa9f2a1449f8f44305726089e3f P 52b3d46fb965452c9ca57b20aad37214: Neither blocks nor log segments found. Creating new log.
I20260928 05:20:06.635896 13753 ts_tablet_manager.cc:1428] T 3a5acaa9f2a1449f8f44305726089e3f P 8b90b255839a41178bd941a6b056134d: Time spent starting tablet: real 0.036s user 0.013s sys 0.002s
I20260928 05:20:06.636301 13598 heartbeater.cc:503] Master 127.0.97.62:37687 was elected leader, sending a full tablet report...
I20260928 05:20:06.636842 13756 log.cc:824] T 3a5acaa9f2a1449f8f44305726089e3f P 52b3d46fb965452c9ca57b20aad37214: Log is configured to *not* fsync() on all Append() calls
I20260928 05:20:06.637156 13755 log.cc:824] T 3a5acaa9f2a1449f8f44305726089e3f P 664827f758474b1da0560894c9c595c5: Log is configured to *not* fsync() on all Append() calls
I20260928 05:20:06.643356 13755 tablet_bootstrap.cc:492] T 3a5acaa9f2a1449f8f44305726089e3f P 664827f758474b1da0560894c9c595c5: No bootstrap required, opened a new log
I20260928 05:20:06.643648 13755 ts_tablet_manager.cc:1397] T 3a5acaa9f2a1449f8f44305726089e3f P 664827f758474b1da0560894c9c595c5: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.003s
I20260928 05:20:06.646871 13756 tablet_bootstrap.cc:492] T 3a5acaa9f2a1449f8f44305726089e3f P 52b3d46fb965452c9ca57b20aad37214: No bootstrap required, opened a new log
I20260928 05:20:06.647166 13756 ts_tablet_manager.cc:1397] T 3a5acaa9f2a1449f8f44305726089e3f P 52b3d46fb965452c9ca57b20aad37214: Time spent bootstrapping tablet: real 0.019s user 0.013s sys 0.000s
I20260928 05:20:06.658795 13756 raft_consensus.cc:348] T 3a5acaa9f2a1449f8f44305726089e3f P 52b3d46fb965452c9ca57b20aad37214 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b90b255839a41178bd941a6b056134d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35721 } } peers { permanent_uuid: "664827f758474b1da0560894c9c595c5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39531 } } peers { permanent_uuid: "52b3d46fb965452c9ca57b20aad37214" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42651 } }
I20260928 05:20:06.659341 13756 raft_consensus.cc:374] T 3a5acaa9f2a1449f8f44305726089e3f P 52b3d46fb965452c9ca57b20aad37214 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:20:06.659492 13756 raft_consensus.cc:729] T 3a5acaa9f2a1449f8f44305726089e3f P 52b3d46fb965452c9ca57b20aad37214 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52b3d46fb965452c9ca57b20aad37214, State: Initialized, Role: FOLLOWER
I20260928 05:20:06.660317 13756 consensus_queue.cc:260] T 3a5acaa9f2a1449f8f44305726089e3f P 52b3d46fb965452c9ca57b20aad37214 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b90b255839a41178bd941a6b056134d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35721 } } peers { permanent_uuid: "664827f758474b1da0560894c9c595c5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39531 } } peers { permanent_uuid: "52b3d46fb965452c9ca57b20aad37214" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42651 } }
I20260928 05:20:06.661572 13755 raft_consensus.cc:348] T 3a5acaa9f2a1449f8f44305726089e3f P 664827f758474b1da0560894c9c595c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b90b255839a41178bd941a6b056134d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35721 } } peers { permanent_uuid: "664827f758474b1da0560894c9c595c5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39531 } } peers { permanent_uuid: "52b3d46fb965452c9ca57b20aad37214" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42651 } }
I20260928 05:20:06.662317 13755 raft_consensus.cc:374] T 3a5acaa9f2a1449f8f44305726089e3f P 664827f758474b1da0560894c9c595c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:20:06.662462 13755 raft_consensus.cc:729] T 3a5acaa9f2a1449f8f44305726089e3f P 664827f758474b1da0560894c9c595c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 664827f758474b1da0560894c9c595c5, State: Initialized, Role: FOLLOWER
I20260928 05:20:06.663166 13755 consensus_queue.cc:260] T 3a5acaa9f2a1449f8f44305726089e3f P 664827f758474b1da0560894c9c595c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b90b255839a41178bd941a6b056134d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35721 } } peers { permanent_uuid: "664827f758474b1da0560894c9c595c5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39531 } } peers { permanent_uuid: "52b3d46fb965452c9ca57b20aad37214" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42651 } }
I20260928 05:20:06.668613 13756 ts_tablet_manager.cc:1428] T 3a5acaa9f2a1449f8f44305726089e3f P 52b3d46fb965452c9ca57b20aad37214: Time spent starting tablet: real 0.021s user 0.010s sys 0.008s
I20260928 05:20:06.669027 13454 heartbeater.cc:503] Master 127.0.97.62:37687 was elected leader, sending a full tablet report...
I20260928 05:20:06.674314 13732 heartbeater.cc:503] Master 127.0.97.62:37687 was elected leader, sending a full tablet report...
I20260928 05:20:06.675052 13755 ts_tablet_manager.cc:1428] T 3a5acaa9f2a1449f8f44305726089e3f P 664827f758474b1da0560894c9c595c5: Time spent starting tablet: real 0.031s user 0.014s sys 0.004s
I20260928 05:20:06.712042 13760 raft_consensus.cc:482] T 3a5acaa9f2a1449f8f44305726089e3f P 52b3d46fb965452c9ca57b20aad37214 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:20:06.712348 13760 raft_consensus.cc:504] T 3a5acaa9f2a1449f8f44305726089e3f P 52b3d46fb965452c9ca57b20aad37214 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b90b255839a41178bd941a6b056134d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35721 } } peers { permanent_uuid: "664827f758474b1da0560894c9c595c5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39531 } } peers { permanent_uuid: "52b3d46fb965452c9ca57b20aad37214" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42651 } }
I20260928 05:20:06.714504 13760 leader_election.cc:288] T 3a5acaa9f2a1449f8f44305726089e3f P 52b3d46fb965452c9ca57b20aad37214 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b90b255839a41178bd941a6b056134d (127.0.97.2:35721), 664827f758474b1da0560894c9c595c5 (127.0.97.3:39531)
I20260928 05:20:06.722393 13749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:06.715312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50933 (local address 127.0.97.2:35721)
0928 05:20:06.715502 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 05:20:06.715508 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:20:06.716052 (+ 544us) server_negotiation.cc:407] Connection header received
0928 05:20:06.716370 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:20:06.716376 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:20:06.716469 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:20:06.716525 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:20:06.717166 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:06.718193 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:06.720603 (+ 2410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:06.721465 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:06.721596 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:06.721691 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 05:20:06.722008 (+ 317us) server_negotiation.cc:299] Negotiation successful
0928 05:20:06.722135 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 05:20:06.722419 13764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:06.715248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35721 (local address 127.0.97.1:50933)
0928 05:20:06.715928 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:06.715948 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 05:20:06.716174 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:20:06.716683 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:20:06.716686 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:20:06.716708 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:20:06.717035 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:20:06.717044 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:20:06.718345 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:20:06.718353 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:20:06.720443 (+ 2090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:20:06.720457 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:20:06.721668 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:20:06.721673 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:20:06.721846 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:06.721878 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 05:20:06.721975 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 05:20:06.722096 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":1077,"threads_started":1}
I20260928 05:20:06.723626 13751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:06.716149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35779 (local address 127.0.97.3:39531)
0928 05:20:06.716324 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 05:20:06.716329 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:20:06.717552 (+ 1223us) server_negotiation.cc:407] Connection header received
0928 05:20:06.717780 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:20:06.717785 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:20:06.717856 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:20:06.717912 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:20:06.718688 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:06.719731 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:06.721576 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:06.722622 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:06.722730 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:06.722822 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 05:20:06.723273 (+ 451us) server_negotiation.cc:299] Negotiation successful
0928 05:20:06.723405 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 05:20:06.723651 13765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:06.715819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39531 (local address 127.0.97.1:35779)
0928 05:20:06.717463 (+ 1644us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:06.717479 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 05:20:06.717644 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:20:06.718073 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:20:06.718077 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:20:06.718099 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:20:06.718495 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:20:06.718509 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:20:06.719901 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:20:06.719906 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:20:06.721433 (+ 1527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:20:06.721445 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:20:06.722973 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:20:06.722980 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:20:06.723128 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:06.723150 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 05:20:06.723230 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 05:20:06.723347 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1532,"thread_start_us":2688,"threads_started":1}
I20260928 05:20:06.724684 13553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a5acaa9f2a1449f8f44305726089e3f" candidate_uuid: "52b3d46fb965452c9ca57b20aad37214" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b90b255839a41178bd941a6b056134d" is_pre_election: true
I20260928 05:20:06.724689 13687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a5acaa9f2a1449f8f44305726089e3f" candidate_uuid: "52b3d46fb965452c9ca57b20aad37214" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "664827f758474b1da0560894c9c595c5" is_pre_election: true
I20260928 05:20:06.725258 13687 raft_consensus.cc:2454] T 3a5acaa9f2a1449f8f44305726089e3f P 664827f758474b1da0560894c9c595c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52b3d46fb965452c9ca57b20aad37214 in term 0.
I20260928 05:20:06.725258 13553 raft_consensus.cc:2454] T 3a5acaa9f2a1449f8f44305726089e3f P 8b90b255839a41178bd941a6b056134d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52b3d46fb965452c9ca57b20aad37214 in term 0.
I20260928 05:20:06.725957 13343 leader_election.cc:302] T 3a5acaa9f2a1449f8f44305726089e3f P 52b3d46fb965452c9ca57b20aad37214 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52b3d46fb965452c9ca57b20aad37214, 8b90b255839a41178bd941a6b056134d; no voters:
I20260928 05:20:06.726416 13760 raft_consensus.cc:2781] T 3a5acaa9f2a1449f8f44305726089e3f P 52b3d46fb965452c9ca57b20aad37214 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 05:20:06.726573 13760 raft_consensus.cc:482] T 3a5acaa9f2a1449f8f44305726089e3f P 52b3d46fb965452c9ca57b20aad37214 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 05:20:06.726661 13760 raft_consensus.cc:3037] T 3a5acaa9f2a1449f8f44305726089e3f P 52b3d46fb965452c9ca57b20aad37214 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:20:06.728729 13760 raft_consensus.cc:504] T 3a5acaa9f2a1449f8f44305726089e3f P 52b3d46fb965452c9ca57b20aad37214 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b90b255839a41178bd941a6b056134d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35721 } } peers { permanent_uuid: "664827f758474b1da0560894c9c595c5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39531 } } peers { permanent_uuid: "52b3d46fb965452c9ca57b20aad37214" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42651 } }
I20260928 05:20:06.729302 13760 leader_election.cc:288] T 3a5acaa9f2a1449f8f44305726089e3f P 52b3d46fb965452c9ca57b20aad37214 [CANDIDATE]: Term 1 election: Requested vote from peers 8b90b255839a41178bd941a6b056134d (127.0.97.2:35721), 664827f758474b1da0560894c9c595c5 (127.0.97.3:39531)
I20260928 05:20:06.729664 13553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a5acaa9f2a1449f8f44305726089e3f" candidate_uuid: "52b3d46fb965452c9ca57b20aad37214" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b90b255839a41178bd941a6b056134d"
I20260928 05:20:06.729781 13687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a5acaa9f2a1449f8f44305726089e3f" candidate_uuid: "52b3d46fb965452c9ca57b20aad37214" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "664827f758474b1da0560894c9c595c5"
I20260928 05:20:06.729861 13553 raft_consensus.cc:3037] T 3a5acaa9f2a1449f8f44305726089e3f P 8b90b255839a41178bd941a6b056134d [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:20:06.729974 13687 raft_consensus.cc:3037] T 3a5acaa9f2a1449f8f44305726089e3f P 664827f758474b1da0560894c9c595c5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:20:06.732908 13687 raft_consensus.cc:2454] T 3a5acaa9f2a1449f8f44305726089e3f P 664827f758474b1da0560894c9c595c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52b3d46fb965452c9ca57b20aad37214 in term 1.
I20260928 05:20:06.732901 13553 raft_consensus.cc:2454] T 3a5acaa9f2a1449f8f44305726089e3f P 8b90b255839a41178bd941a6b056134d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52b3d46fb965452c9ca57b20aad37214 in term 1.
I20260928 05:20:06.733237 13343 leader_election.cc:302] T 3a5acaa9f2a1449f8f44305726089e3f P 52b3d46fb965452c9ca57b20aad37214 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52b3d46fb965452c9ca57b20aad37214, 8b90b255839a41178bd941a6b056134d; no voters:
I20260928 05:20:06.733594 13760 raft_consensus.cc:2781] T 3a5acaa9f2a1449f8f44305726089e3f P 52b3d46fb965452c9ca57b20aad37214 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:20:06.734127 13760 raft_consensus.cc:686] T 3a5acaa9f2a1449f8f44305726089e3f P 52b3d46fb965452c9ca57b20aad37214 [term 1 LEADER]: Becoming Leader. State: Replica: 52b3d46fb965452c9ca57b20aad37214, State: Running, Role: LEADER
I20260928 05:20:06.734727 13760 consensus_queue.cc:237] T 3a5acaa9f2a1449f8f44305726089e3f P 52b3d46fb965452c9ca57b20aad37214 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b90b255839a41178bd941a6b056134d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35721 } } peers { permanent_uuid: "664827f758474b1da0560894c9c595c5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39531 } } peers { permanent_uuid: "52b3d46fb965452c9ca57b20aad37214" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42651 } }
I20260928 05:20:06.739243 13256 catalog_manager.cc:5495] T 3a5acaa9f2a1449f8f44305726089e3f P 52b3d46fb965452c9ca57b20aad37214 reported cstate change: term changed from 0 to 1, leader changed from <none> to 52b3d46fb965452c9ca57b20aad37214 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "52b3d46fb965452c9ca57b20aad37214" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b90b255839a41178bd941a6b056134d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "664827f758474b1da0560894c9c595c5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52b3d46fb965452c9ca57b20aad37214" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42651 } health_report { overall_health: HEALTHY } } }
I20260928 05:20:06.774530 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:06.769892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35794 (local address 127.0.97.1:42651)
0928 05:20:06.770070 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 05:20:06.770074 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:20:06.770091 (+ 17us) server_negotiation.cc:407] Connection header received
0928 05:20:06.770227 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:20:06.770231 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:20:06.770303 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:20:06.770423 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:20:06.770686 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:06.771383 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:06.772657 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:06.773075 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:06.773164 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:06.773561 (+ 397us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:20:06.774050 (+ 489us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:20:06.774148 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 05:20:06.774220 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 05:20:06.774278 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 05:20:06.781888 13687 raft_consensus.cc:1261] T 3a5acaa9f2a1449f8f44305726089e3f P 664827f758474b1da0560894c9c595c5 [term 1 FOLLOWER]: Refusing update from remote peer 52b3d46fb965452c9ca57b20aad37214: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:20:06.781894 13553 raft_consensus.cc:1261] T 3a5acaa9f2a1449f8f44305726089e3f P 8b90b255839a41178bd941a6b056134d [term 1 FOLLOWER]: Refusing update from remote peer 52b3d46fb965452c9ca57b20aad37214: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:20:06.782716 13771 consensus_queue.cc:1035] T 3a5acaa9f2a1449f8f44305726089e3f P 52b3d46fb965452c9ca57b20aad37214 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b90b255839a41178bd941a6b056134d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:20:06.783051 13760 consensus_queue.cc:1035] T 3a5acaa9f2a1449f8f44305726089e3f P 52b3d46fb965452c9ca57b20aad37214 [LEADER]: Connected to new peer: Peer: permanent_uuid: "664827f758474b1da0560894c9c595c5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:20:06.800319 13776 mvcc.cc:205] Tried to move back new op lower bound from 7334186216569253888 to 7334186216389660672. Current Snapshot: MvccSnapshot[applied={T|T < 7334186216569253888}]
I20260928 05:20:06.800519 13775 mvcc.cc:205] Tried to move back new op lower bound from 7334186216569253888 to 7334186216389660672. Current Snapshot: MvccSnapshot[applied={T|T < 7334186216569253888}]
I20260928 05:20:06.805032 13255 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40292:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:20:06.805280 13255 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40292:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:20:06.809060 13255 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 30e4edfa8186490b8cb4258d9524eae7: Sending DeleteTablet for 3 replicas of tablet 3a5acaa9f2a1449f8f44305726089e3f
I20260928 05:20:06.809919 13533 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a5acaa9f2a1449f8f44305726089e3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:20:06 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:45912
I20260928 05:20:06.810035 13667 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a5acaa9f2a1449f8f44305726089e3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:20:06 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37778
I20260928 05:20:06.810467 13532 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a5acaa9f2a1449f8f44305726089e3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:20:06 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:45912
I20260928 05:20:06.810734 13783 tablet_replica.cc:330] T 3a5acaa9f2a1449f8f44305726089e3f P 8b90b255839a41178bd941a6b056134d: stopping tablet replica
I20260928 05:20:06.811076 13783 raft_consensus.cc:2229] T 3a5acaa9f2a1449f8f44305726089e3f P 8b90b255839a41178bd941a6b056134d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:20:06.811106 13389 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a5acaa9f2a1449f8f44305726089e3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:20:06 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35786
I20260928 05:20:06.811321 13784 tablet_replica.cc:330] T 3a5acaa9f2a1449f8f44305726089e3f P 664827f758474b1da0560894c9c595c5: stopping tablet replica
I20260928 05:20:06.811501 13783 raft_consensus.cc:2258] T 3a5acaa9f2a1449f8f44305726089e3f P 8b90b255839a41178bd941a6b056134d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:20:06.811741 13784 raft_consensus.cc:2229] T 3a5acaa9f2a1449f8f44305726089e3f P 664827f758474b1da0560894c9c595c5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:20:06.812211 13784 raft_consensus.cc:2258] T 3a5acaa9f2a1449f8f44305726089e3f P 664827f758474b1da0560894c9c595c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:20:06.814034 13784 ts_tablet_manager.cc:1905] T 3a5acaa9f2a1449f8f44305726089e3f P 664827f758474b1da0560894c9c595c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:20:06.818764 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 13293
I20260928 05:20:06.820577 13784 ts_tablet_manager.cc:1918] T 3a5acaa9f2a1449f8f44305726089e3f P 664827f758474b1da0560894c9c595c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:20:06.820789 13784 log.cc:1196] T 3a5acaa9f2a1449f8f44305726089e3f P 664827f758474b1da0560894c9c595c5: Deleting WAL directory at /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ts-2/wal/wals/3a5acaa9f2a1449f8f44305726089e3f
I20260928 05:20:06.821370 13784 ts_tablet_manager.cc:1939] T 3a5acaa9f2a1449f8f44305726089e3f P 664827f758474b1da0560894c9c595c5: Deleting consensus metadata
I20260928 05:20:06.823017 13243 catalog_manager.cc:4841] TS 664827f758474b1da0560894c9c595c5 (127.0.97.3:39531): tablet 3a5acaa9f2a1449f8f44305726089e3f (table test-table [id=e157fea335f14be980b488b03c3e3223]) successfully deleted
I20260928 05:20:06.824747 13783 ts_tablet_manager.cc:1905] T 3a5acaa9f2a1449f8f44305726089e3f P 8b90b255839a41178bd941a6b056134d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:20:06.825225 13785 tablet_replica.cc:330] T 3a5acaa9f2a1449f8f44305726089e3f P 52b3d46fb965452c9ca57b20aad37214: stopping tablet replica
I20260928 05:20:06.825469 13785 raft_consensus.cc:2229] T 3a5acaa9f2a1449f8f44305726089e3f P 52b3d46fb965452c9ca57b20aad37214 [term 1 LEADER]: Raft consensus shutting down.
I20260928 05:20:06.825927 13785 raft_consensus.cc:2258] T 3a5acaa9f2a1449f8f44305726089e3f P 52b3d46fb965452c9ca57b20aad37214 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:20:06.827504 13785 ts_tablet_manager.cc:1905] T 3a5acaa9f2a1449f8f44305726089e3f P 52b3d46fb965452c9ca57b20aad37214: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:20:06.831086 13783 ts_tablet_manager.cc:1918] T 3a5acaa9f2a1449f8f44305726089e3f P 8b90b255839a41178bd941a6b056134d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:20:06.831240 13783 log.cc:1196] T 3a5acaa9f2a1449f8f44305726089e3f P 8b90b255839a41178bd941a6b056134d: Deleting WAL directory at /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ts-1/wal/wals/3a5acaa9f2a1449f8f44305726089e3f
I20260928 05:20:06.831732 13783 ts_tablet_manager.cc:1939] T 3a5acaa9f2a1449f8f44305726089e3f P 8b90b255839a41178bd941a6b056134d: Deleting consensus metadata
I20260928 05:20:06.833359 13242 catalog_manager.cc:4841] TS 8b90b255839a41178bd941a6b056134d (127.0.97.2:35721): tablet 3a5acaa9f2a1449f8f44305726089e3f (table test-table [id=e157fea335f14be980b488b03c3e3223]) successfully deleted
W20260928 05:20:06.834295 13242 catalog_manager.cc:4808] TS 8b90b255839a41178bd941a6b056134d (127.0.97.2:35721): delete failed for tablet 3a5acaa9f2a1449f8f44305726089e3f because the tablet was not found. No further retry: Not found: Tablet not found: 3a5acaa9f2a1449f8f44305726089e3f
I20260928 05:20:06.834622 13785 ts_tablet_manager.cc:1918] T 3a5acaa9f2a1449f8f44305726089e3f P 52b3d46fb965452c9ca57b20aad37214: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:20:06.834789 13785 log.cc:1196] T 3a5acaa9f2a1449f8f44305726089e3f P 52b3d46fb965452c9ca57b20aad37214: Deleting WAL directory at /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572622576160-388-0/minicluster-data/ts-0/wal/wals/3a5acaa9f2a1449f8f44305726089e3f
I20260928 05:20:06.835284 13785 ts_tablet_manager.cc:1939] T 3a5acaa9f2a1449f8f44305726089e3f P 52b3d46fb965452c9ca57b20aad37214: Deleting consensus metadata
I20260928 05:20:06.836756 13243 catalog_manager.cc:4841] TS 52b3d46fb965452c9ca57b20aad37214 (127.0.97.1:42651): tablet 3a5acaa9f2a1449f8f44305726089e3f (table test-table [id=e157fea335f14be980b488b03c3e3223]) successfully deleted
I20260928 05:20:06.838626 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:06.820069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35808 (local address 127.0.97.1:42651)
0928 05:20:06.820216 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 05:20:06.820221 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:20:06.820239 (+ 18us) server_negotiation.cc:407] Connection header received
0928 05:20:06.820289 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:20:06.820294 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:20:06.820384 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:20:06.820490 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:20:06.823374 (+ 2884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:06.824522 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:06.826871 (+ 2349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:06.827517 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:06.827635 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:06.831490 (+ 3855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:20:06.831503 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:20:06.831519 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:20:06.831575 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:20:06.835513 (+ 3938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:06.836160 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:06.836166 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:06.836168 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:06.836321 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:06.836697 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:06.836702 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:06.836705 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:06.837271 (+ 566us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:20:06.837559 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0928 05:20:06.838086 (+ 527us) server_negotiation.cc:299] Negotiation successful
0928 05:20:06.838297 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 05:20:06.839152 13449 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:20:06.931825 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 13293
I20260928 05:20:06.945078 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 13458
I20260928 05:20:06.957158 13749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:06.945703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45922 (local address 127.0.97.2:35721)
0928 05:20:06.945873 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 05:20:06.945877 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:20:06.945902 (+ 25us) server_negotiation.cc:407] Connection header received
0928 05:20:06.946032 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:20:06.946036 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:20:06.946111 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:20:06.946196 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:20:06.947295 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:06.947972 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:06.949214 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:06.949638 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:06.949759 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:06.951977 (+ 2218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:20:06.951991 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:20:06.952006 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:20:06.952039 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:20:06.954943 (+ 2904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:06.955476 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:06.955482 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:06.955485 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:06.955612 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:06.955900 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:06.955903 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:06.955906 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:06.956250 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:20:06.956453 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0928 05:20:06.956673 (+ 220us) server_negotiation.cc:299] Negotiation successful
0928 05:20:06.956816 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 05:20:06.957731 13593 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:20:07.043550 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 13458
I20260928 05:20:07.056800 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 13602
I20260928 05:20:07.068672 13751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:07.057541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37788 (local address 127.0.97.3:39531)
0928 05:20:07.057707 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 05:20:07.057711 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:20:07.057730 (+ 19us) server_negotiation.cc:407] Connection header received
0928 05:20:07.057845 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:20:07.057849 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:20:07.057921 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:20:07.058007 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:20:07.059175 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:07.059867 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:07.061123 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:07.061521 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:07.061608 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:07.063801 (+ 2193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:20:07.063815 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:20:07.063831 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:20:07.063861 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:20:07.066535 (+ 2674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:07.067095 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:07.067102 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:07.067104 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:07.067209 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:07.067453 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:07.067456 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:07.067458 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:07.067799 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:20:07.068031 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0928 05:20:07.068232 (+ 201us) server_negotiation.cc:299] Negotiation successful
0928 05:20:07.068382 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 05:20:07.069489 13727 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:20:07.153187 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 13602
I20260928 05:20:07.166330 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 13223
I20260928 05:20:07.171274 13286 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:20:07.276592 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 13223
I20260928 05:20:07.293306 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 05:20:07.627002 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 05:20:07.627195 388 mini_ranger.cc:66] Stopping Ranger...
I20260928 05:20:07.960918 388 mini_ranger.cc:68] Stopped Ranger
2026-09-28 05:20:07.961 UTC [12276] LOG: received smart shutdown request
2026-09-28 05:20:07.964 UTC [12276] LOG: background worker "logical replication launcher" (PID 12285) exited with exit code 1
2026-09-28 05:20:07.965 UTC [12280] LOG: shutting down
2026-09-28 05:20:08.101 UTC [12276] LOG: database system is shut down
2026-09-28T05:20:08Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (49384 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572622576160-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:20:08 dist-test-slave-whsn krb5kdc[13798](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:20:08 dist-test-slave-whsn krb5kdc[13798](info): set up 2 sockets
Sep 28 05:20:08 dist-test-slave-whsn krb5kdc[13798](info): commencing operation
krb5kdc: starting...
W20260928 05:20:09.212688 388 mini_kdc.cc:121] Time spent starting KDC: real 0.977s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572622576160-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572622576160-388-0/krb5kdc/test-user.keytab.
Sep 28 05:20:09 dist-test-slave-whsn krb5kdc[13798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572809, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:20:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:20:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 05:20:09.403827 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572622576160-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572622576160-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572622576160-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35731
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35731,127.0.97.61:44427,127.0.97.60:41507
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:20:09.704438 13811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:20:09.704696 13811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:20:09.704823 13811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:20:09.711603 13811 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:20:09.711699 13811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:20:09.711800 13811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:20:09.711885 13811 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:20:09.711953 13811 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:20:09.721129 13811 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572622576160-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572622576160-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35731,127.0.97.61:44427,127.0.97.60:41507
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35731
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572622576160-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:20:09.722268 13811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:20:09.723489 13811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:20:09.734036 13817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:20:09.734100 13816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:20:09.734133 13819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:20:09.735034 13811 server_base.cc:1034] running on GCE node
I20260928 05:20:09.735728 13811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:20:09.736999 13811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:20:09.738195 13811 hybrid_clock.cc:648] HybridClock initialized: now 1790572809738152 us; error 38 us; skew 500 ppm
I20260928 05:20:09.738499 13811 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:20:09 dist-test-slave-whsn krb5kdc[13798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572809, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:20:09.741206 13811 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 05:20:09.742622 13811 webserver.cc:460] Webserver started at http://127.0.97.62:42421/ using document root <none> and password file <none>
I20260928 05:20:09.743402 13811 fs_manager.cc:359] Metadata directory not provided
I20260928 05:20:09.743520 13811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:20:09.743816 13811 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:20:09.747534 13811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572622576160-388-0/minicluster-data/master-0/data/instance:
uuid: "f648f22b0d60437394cec07e21635a30"
format_stamp: "Formatted at 2026-09-28 05:20:09 on dist-test-slave-whsn"
server_key: "2361bf7db2a7375f2200797e11dd51ec"
server_key_iv: "03a476929710e641600e5ae5400562a8"
server_key_version: "encryptionkey@0"
I20260928 05:20:09.748396 13811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572622576160-388-0/minicluster-data/master-0/wal/instance:
uuid: "f648f22b0d60437394cec07e21635a30"
format_stamp: "Formatted at 2026-09-28 05:20:09 on dist-test-slave-whsn"
server_key: "2361bf7db2a7375f2200797e11dd51ec"
server_key_iv: "03a476929710e641600e5ae5400562a8"
server_key_version: "encryptionkey@0"
I20260928 05:20:09.753660 13811 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260928 05:20:09.756681 13826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:20:09.757902 13811 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 05:20:09.758049 13811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572622576160-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572622576160-388-0/minicluster-data/master-0/wal
uuid: "f648f22b0d60437394cec07e21635a30"
format_stamp: "Formatted at 2026-09-28 05:20:09 on dist-test-slave-whsn"
server_key: "2361bf7db2a7375f2200797e11dd51ec"
server_key_iv: "03a476929710e641600e5ae5400562a8"
server_key_version: "encryptionkey@0"
I20260928 05:20:09.758212 13811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572622576160-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572622576160-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572622576160-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:20:09.780361 13811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:20:09.783807 13811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:20:09.784044 13811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:20:09.800372 13811 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35731
I20260928 05:20:09.800418 13877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35731 every 8 connection(s)
I20260928 05:20:09.801586 13811 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:20:09.804903 13878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:20:09.810284 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 13811
I20260928 05:20:09.810475 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572622576160-388-0/minicluster-data/master-0/wal/instance
I20260928 05:20:09.811043 388 external_mini_cluster.cc:1428] Setting key 094b9557988d1d75082a53543bf77bc6
I20260928 05:20:09.810926 13878 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35731 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 05:20:09 dist-test-slave-whsn krb5kdc[13798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572809, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260928 05:20:09.830525 13880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:09.813736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45572 (local address 127.0.97.62:35731)
0928 05:20:09.814457 (+ 721us) server_negotiation.cc:206] Beginning negotiation
0928 05:20:09.814470 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 05:20:09.814881 (+ 411us) server_negotiation.cc:407] Connection header received
0928 05:20:09.816689 (+ 1808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:20:09.816694 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:20:09.816930 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:20:09.817100 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:20:09.818254 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:09.819376 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:09.820674 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:09.821248 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:09.821416 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:09.824400 (+ 2984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:20:09.824416 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:20:09.824426 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:20:09.824452 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:20:09.826942 (+ 2490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:09.827532 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:09.827538 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:09.827540 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:09.827667 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:09.828027 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:09.828032 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:09.828035 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:09.828458 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:20:09.828620 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 05:20:09.829182 (+ 562us) server_negotiation.cc:299] Negotiation successful
0928 05:20:09.829455 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":508,"thread_start_us":326,"threads_started":1}
I20260928 05:20:09.830566 13881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:09.813737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35731 (local address 127.0.0.1:45572)
0928 05:20:09.814745 (+ 1008us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:09.814776 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0928 05:20:09.816366 (+ 1590us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:20:09.817138 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:20:09.817144 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:20:09.817772 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:20:09.818077 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:20:09.818094 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:20:09.819519 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:20:09.819522 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:20:09.820549 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:20:09.820561 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:20:09.821469 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:20:09.821477 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:20:09.821668 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:09.822232 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:20:09.822252 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:20:09.824206 (+ 1954us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:20:09.827053 (+ 2847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:20:09.827063 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:20:09.827073 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:20:09.827420 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:20:09.827774 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:20:09.827779 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:20:09.827783 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:20:09.827933 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:20:09.828611 (+ 678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:20:09.828619 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:20:09.828810 (+ 191us) client_negotiation.cc:769] Sending connection context
0928 05:20:09.829054 (+ 244us) client_negotiation.cc:240] Negotiation successful
0928 05:20:09.829581 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":804,"mutex_wait_us":133,"spinlock_wait_cycles":23296,"thread_start_us":638,"threads_started":1}
I20260928 05:20:09.833220 13878 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44427 } has no permanent_uuid. Determining permanent_uuid...
I20260928 05:20:09.834120 13881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:09.833786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44427 (local address 127.0.0.1:33396)
0928 05:20:09.833928 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:09.834022 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260928 05:20:09.834323 13827 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44427: connect: Connection refused (error 111)
I20260928 05:20:09.835744 13881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:09.835385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44427 (local address 127.0.0.1:33398)
0928 05:20:09.835520 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:09.835609 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260928 05:20:09.836009 13878 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:44427: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44427: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572622576160-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572622576160-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260928 05:20:09.851698 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572622576160-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572622576160-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572622576160-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572622576160-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:44427
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572622576160-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35731,127.0.97.61:44427,127.0.97.60:41507
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 05:20:09.885303 13878 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44427 } attempt: 1
I20260928 05:20:09.886332 13881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:09.885939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44427 (local address 127.0.0.1:33406)
0928 05:20:09.886115 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:09.886208 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260928 05:20:09.887387 13881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:09.886992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44427 (local address 127.0.0.1:33420)
0928 05:20:09.887146 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:09.887228 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260928 05:20:09.887603 13878 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:44427: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44427: connect: Connection refused (error 111)
I20260928 05:20:09.955837 13878 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44427 } attempt: 2
I20260928 05:20:09.957020 13881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:09.956441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44427 (local address 127.0.0.1:33428)
0928 05:20:09.956710 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:09.956824 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260928 05:20:09.957994 13881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:09.957618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44427 (local address 127.0.0.1:33436)
0928 05:20:09.957762 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:09.957844 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260928 05:20:09.958199 13878 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:44427: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44427: connect: Connection refused (error 111)
I20260928 05:20:10.049480 13878 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44427 } attempt: 3
I20260928 05:20:10.050721 13881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:10.050191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44427 (local address 127.0.0.1:33442)
0928 05:20:10.050437 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:10.050547 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260928 05:20:10.051816 13881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:10.051411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44427 (local address 127.0.0.1:33456)
0928 05:20:10.051568 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:10.051657 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260928 05:20:10.052037 13878 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:44427: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44427: connect: Connection refused (error 111)
W20260928 05:20:10.152431 13885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:20:10.152685 13885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:20:10.152766 13885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:20:10.159628 13885 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:20:10.159722 13885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:20:10.159773 13885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:20:10.159828 13885 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:20:10.159878 13885 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:20:10.169170 13885 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572622576160-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572622576160-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35731,127.0.97.61:44427,127.0.97.60:41507
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572622576160-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:44427
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572622576160-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572622576160-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:20:10.170297 13885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:20:10.171466 13885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:20:10.181934 13890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:20:10.181964 13891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:20:10.182003 13893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:20:10.182978 13885 server_base.cc:1034] running on GCE node
I20260928 05:20:10.183604 13885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:20:10.184787 13885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:20:10.185993 13885 hybrid_clock.cc:648] HybridClock initialized: now 1790572810185945 us; error 42 us; skew 500 ppm
I20260928 05:20:10.186304 13885 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:20:10 dist-test-slave-whsn krb5kdc[13798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572810, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:20:10.189147 13885 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260928 05:20:10.190496 13885 webserver.cc:460] Webserver started at http://127.0.97.61:39545/ using document root <none> and password file <none>
I20260928 05:20:10.191118 13885 fs_manager.cc:359] Metadata directory not provided
I20260928 05:20:10.191224 13885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:20:10.191485 13885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:20:10.194274 13885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572622576160-388-0/minicluster-data/master-1/data/instance:
uuid: "b5005445bae743bebdde0d6021142b9c"
format_stamp: "Formatted at 2026-09-28 05:20:10 on dist-test-slave-whsn"
server_key: "ab17be2c0c010c34bf1fbf83aca2a07f"
server_key_iv: "ffac34b2f0796516a1edb3ceee558910"
server_key_version: "encryptionkey@0"
I20260928 05:20:10.194988 13885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572622576160-388-0/minicluster-data/master-1/wal/instance:
uuid: "b5005445bae743bebdde0d6021142b9c"
format_stamp: "Formatted at 2026-09-28 05:20:10 on dist-test-slave-whsn"
server_key: "ab17be2c0c010c34bf1fbf83aca2a07f"
server_key_iv: "ffac34b2f0796516a1edb3ceee558910"
server_key_version: "encryptionkey@0"
I20260928 05:20:10.195246 13878 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44427 } attempt: 4
I20260928 05:20:10.196319 13881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:10.195815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44427 (local address 127.0.0.1:33472)
0928 05:20:10.196108 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:10.196196 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145}
I20260928 05:20:10.197259 13881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:10.196938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44427 (local address 127.0.0.1:33482)
0928 05:20:10.197070 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:10.197146 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260928 05:20:10.197415 13878 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:44427: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44427: connect: Connection refused (error 111)
I20260928 05:20:10.200120 13885 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260928 05:20:10.203285 13900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:20:10.204492 13885 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 05:20:10.204630 13885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572622576160-388-0/minicluster-data/master-1/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572622576160-388-0/minicluster-data/master-1/wal
uuid: "b5005445bae743bebdde0d6021142b9c"
format_stamp: "Formatted at 2026-09-28 05:20:10 on dist-test-slave-whsn"
server_key: "ab17be2c0c010c34bf1fbf83aca2a07f"
server_key_iv: "ffac34b2f0796516a1edb3ceee558910"
server_key_version: "encryptionkey@0"
I20260928 05:20:10.204793 13885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572622576160-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572622576160-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572622576160-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:20:10.220412 13885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:20:10.224009 13885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:20:10.224242 13885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:20:10.240716 13885 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:44427
I20260928 05:20:10.240751 13951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:44427 every 8 connection(s)
I20260928 05:20:10.242007 13885 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572622576160-388-0/minicluster-data/master-1/data/info.pb
I20260928 05:20:10.245435 13952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:20:10.247064 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 13885
I20260928 05:20:10.247254 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572622576160-388-0/minicluster-data/master-1/wal/instance
I20260928 05:20:10.247763 388 external_mini_cluster.cc:1428] Setting key 813d9406262b261e953595a986888a55
I20260928 05:20:10.253808 13952 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35731 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 05:20:10 dist-test-slave-whsn krb5kdc[13798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572810, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
I20260928 05:20:10.273140 13880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:10.257052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45580 (local address 127.0.97.62:35731)
0928 05:20:10.257214 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 05:20:10.257219 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:20:10.258551 (+ 1332us) server_negotiation.cc:407] Connection header received
0928 05:20:10.260308 (+ 1757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:20:10.260314 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:20:10.260384 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:20:10.260479 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:20:10.262028 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:10.262754 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:10.264196 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:10.264596 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:10.264702 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:10.267793 (+ 3091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:20:10.267799 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:20:10.267803 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:20:10.267842 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:20:10.270462 (+ 2620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:10.271086 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:10.271090 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:10.271092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:10.271199 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:10.271577 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:10.271582 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:10.271585 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:10.271835 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:20:10.271974 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 05:20:10.272589 (+ 615us) server_negotiation.cc:299] Negotiation successful
0928 05:20:10.272798 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 05:20:10.273964 13955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:10.257653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35731 (local address 127.0.0.1:45580)
0928 05:20:10.258400 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:10.258439 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 05:20:10.260040 (+ 1601us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:20:10.260711 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:20:10.260719 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:20:10.261337 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:20:10.261862 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:20:10.261879 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:20:10.262897 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:20:10.262901 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:20:10.264063 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:20:10.264078 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:20:10.264794 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:20:10.264801 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:20:10.265034 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:10.265623 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:20:10.265644 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:20:10.267595 (+ 1951us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:20:10.270586 (+ 2991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:20:10.270595 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:20:10.270609 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:20:10.270972 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:20:10.271293 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:20:10.271298 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:20:10.271302 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:20:10.271469 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:20:10.271950 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:20:10.271957 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:20:10.272245 (+ 288us) client_negotiation.cc:769] Sending connection context
0928 05:20:10.272470 (+ 225us) client_negotiation.cc:240] Negotiation successful
0928 05:20:10.272923 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":331,"threads_started":1}
I20260928 05:20:10.275699 13952 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44427 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 05:20:10 dist-test-slave-whsn krb5kdc[13798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572810, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260928 05:20:10.290455 13955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:10.276486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44427 (local address 127.0.0.1:33498)
0928 05:20:10.276679 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:10.276697 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 05:20:10.276786 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:20:10.277787 (+ 1001us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:20:10.277792 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:20:10.278206 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:20:10.278374 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:20:10.278384 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:20:10.280014 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:20:10.280019 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:20:10.281271 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:20:10.281285 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:20:10.282209 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:20:10.282215 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:20:10.282379 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:10.282928 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:20:10.282948 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:20:10.285322 (+ 2374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:20:10.288234 (+ 2912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:20:10.288238 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:20:10.288240 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:20:10.288580 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:20:10.288986 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:20:10.288991 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:20:10.288994 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:20:10.289159 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:20:10.289797 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:20:10.289802 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:20:10.289925 (+ 123us) client_negotiation.cc:769] Sending connection context
0928 05:20:10.290035 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 05:20:10.290212 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260928 05:20:10.290809 13957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:10.276687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33498 (local address 127.0.97.61:44427)
0928 05:20:10.277390 (+ 703us) server_negotiation.cc:206] Beginning negotiation
0928 05:20:10.277396 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:20:10.277422 (+ 26us) server_negotiation.cc:407] Connection header received
0928 05:20:10.277465 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:20:10.277470 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:20:10.277636 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:20:10.277753 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:20:10.278568 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:10.279843 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:10.281411 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:10.282040 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:10.282143 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:10.285521 (+ 3378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:20:10.285532 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:20:10.285544 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:20:10.285572 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:20:10.288133 (+ 2561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:10.288684 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:10.288691 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:10.288693 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:10.288803 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:10.289257 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:10.289260 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:10.289262 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:10.289634 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:20:10.289816 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0928 05:20:10.290154 (+ 338us) server_negotiation.cc:299] Negotiation successful
0928 05:20:10.290402 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":395,"threads_started":1}
I20260928 05:20:10.293160 13952 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41507 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572622576160-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572622576160-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260928 05:20:10.294214 13955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:10.293741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41507 (local address 127.0.0.1:47590)
0928 05:20:10.293914 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:10.294059 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260928 05:20:10.294446 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572622576160-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572622576160-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572622576160-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572622576160-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:41507
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572622576160-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35731,127.0.97.61:44427,127.0.97.60:41507
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:20:10.294463 13903 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41507: connect: Connection refused (error 111)
I20260928 05:20:10.295879 13955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:10.295541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41507 (local address 127.0.0.1:47598)
0928 05:20:10.295690 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:10.295773 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260928 05:20:10.296082 13952 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41507: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41507: connect: Connection refused (error 111)
I20260928 05:20:10.345372 13952 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41507 } attempt: 1
I20260928 05:20:10.346616 13955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:10.346108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41507 (local address 127.0.0.1:47606)
0928 05:20:10.346330 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:10.346445 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260928 05:20:10.347643 13955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:10.347285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41507 (local address 127.0.0.1:47610)
0928 05:20:10.347451 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:10.347535 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260928 05:20:10.347831 13952 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41507: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41507: connect: Connection refused (error 111)
I20260928 05:20:10.416059 13952 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41507 } attempt: 2
I20260928 05:20:10.417186 13955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:10.416726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41507 (local address 127.0.0.1:47612)
0928 05:20:10.416947 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:10.417042 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260928 05:20:10.418192 13955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:10.417795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41507 (local address 127.0.0.1:47618)
0928 05:20:10.417968 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:10.418055 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260928 05:20:10.418402 13952 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41507: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41507: connect: Connection refused (error 111)
I20260928 05:20:10.496640 13878 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44427 } attempt: 5
Sep 28 05:20:10 dist-test-slave-whsn krb5kdc[13798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572809, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260928 05:20:10.509618 13952 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41507 } attempt: 3
I20260928 05:20:10.510627 13955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:10.510144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41507 (local address 127.0.0.1:47630)
0928 05:20:10.510347 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:10.510441 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260928 05:20:10.511662 13955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:10.511240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41507 (local address 127.0.0.1:47638)
0928 05:20:10.511463 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:10.511539 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
W20260928 05:20:10.511909 13952 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41507: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41507: connect: Connection refused (error 111)
I20260928 05:20:10.514757 13881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:10.497369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44427 (local address 127.0.0.1:33508)
0928 05:20:10.497783 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:10.497804 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 05:20:10.497946 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:20:10.498778 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:20:10.498783 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:20:10.499258 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:20:10.499485 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:20:10.499498 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:20:10.500770 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:20:10.500774 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:20:10.502068 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:20:10.502085 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:20:10.503010 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:20:10.503016 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:20:10.503201 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:10.506497 (+ 3296us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:20:10.506522 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:20:10.509230 (+ 2708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:20:10.512756 (+ 3526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:20:10.512760 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:20:10.512762 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:20:10.513161 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:20:10.513497 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:20:10.513499 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:20:10.513502 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:20:10.513621 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:20:10.514136 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:20:10.514139 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:20:10.514256 (+ 117us) client_negotiation.cc:769] Sending connection context
0928 05:20:10.514352 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 05:20:10.514537 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279}
I20260928 05:20:10.514977 13957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:10.498298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33508 (local address 127.0.97.61:44427)
0928 05:20:10.498470 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 05:20:10.498476 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:20:10.498494 (+ 18us) server_negotiation.cc:407] Connection header received
0928 05:20:10.498555 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:20:10.498560 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:20:10.498643 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:20:10.498738 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:20:10.499639 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:10.500635 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:10.502207 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:10.502856 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:10.502977 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:10.509403 (+ 6426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:20:10.509408 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:20:10.509412 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:20:10.509453 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:20:10.512612 (+ 3159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:10.513261 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:10.513266 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:10.513270 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:10.513402 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:10.513707 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:10.513711 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:10.513715 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:10.514045 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:20:10.514183 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 05:20:10.514477 (+ 294us) server_negotiation.cc:299] Negotiation successful
0928 05:20:10.514678 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 05:20:10.515902 13878 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41507 } has no permanent_uuid. Determining permanent_uuid...
I20260928 05:20:10.517005 13881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:10.516565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41507 (local address 127.0.0.1:47642)
0928 05:20:10.516741 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:10.516834 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260928 05:20:10.517940 13881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:10.517594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41507 (local address 127.0.0.1:47654)
0928 05:20:10.517735 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:10.517832 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260928 05:20:10.518122 13878 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41507: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41507: connect: Connection refused (error 111)
I20260928 05:20:10.569324 13878 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41507 } attempt: 1
I20260928 05:20:10.570390 13881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:10.569911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41507 (local address 127.0.0.1:47656)
0928 05:20:10.570094 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:10.570202 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260928 05:20:10.571302 13881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:10.570948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41507 (local address 127.0.0.1:47664)
0928 05:20:10.571099 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:10.571180 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260928 05:20:10.571498 13878 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41507: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41507: connect: Connection refused (error 111)
W20260928 05:20:10.611335 13959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:20:10.611598 13959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:20:10.611688 13959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:20:10.618569 13959 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:20:10.618654 13959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:20:10.618703 13959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:20:10.618755 13959 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:20:10.618796 13959 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:20:10.627854 13959 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572622576160-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572622576160-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35731,127.0.97.61:44427,127.0.97.60:41507
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572622576160-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:41507
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572622576160-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572622576160-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:20:10.628988 13959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:20:10.630158 13959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 05:20:10.639696 13878 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41507 } attempt: 2
I20260928 05:20:10.640980 13881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:10.640354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41507 (local address 127.0.0.1:47670)
0928 05:20:10.640632 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:10.640734 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260928 05:20:10.641305 13964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:20:10.641366 13965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:20:10.641554 13967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:20:10.642398 13881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:10.641957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41507 (local address 127.0.0.1:47686)
0928 05:20:10.642151 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:10.642247 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260928 05:20:10.642590 13959 server_base.cc:1034] running on GCE node
W20260928 05:20:10.642627 13878 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41507: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41507: connect: Connection refused (error 111)
I20260928 05:20:10.643347 13959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:20:10.644645 13959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:20:10.645851 13959 hybrid_clock.cc:648] HybridClock initialized: now 1790572810645770 us; error 64 us; skew 500 ppm
I20260928 05:20:10.646181 13959 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:20:10 dist-test-slave-whsn krb5kdc[13798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572810, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:20:10.648975 13959 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260928 05:20:10.650252 13959 webserver.cc:460] Webserver started at http://127.0.97.60:46547/ using document root <none> and password file <none>
I20260928 05:20:10.650877 13959 fs_manager.cc:359] Metadata directory not provided
I20260928 05:20:10.650985 13959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:20:10.651234 13959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:20:10.653975 13959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572622576160-388-0/minicluster-data/master-2/data/instance:
uuid: "a404cc2bb72145ffbad2de42faa3dafc"
format_stamp: "Formatted at 2026-09-28 05:20:10 on dist-test-slave-whsn"
server_key: "ac8b72635eb13810cf11830f06a23e5d"
server_key_iv: "626052cc7805d5bc8bc33eb073d8e734"
server_key_version: "encryptionkey@0"
I20260928 05:20:10.654665 13959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572622576160-388-0/minicluster-data/master-2/wal/instance:
uuid: "a404cc2bb72145ffbad2de42faa3dafc"
format_stamp: "Formatted at 2026-09-28 05:20:10 on dist-test-slave-whsn"
server_key: "ac8b72635eb13810cf11830f06a23e5d"
server_key_iv: "626052cc7805d5bc8bc33eb073d8e734"
server_key_version: "encryptionkey@0"
I20260928 05:20:10.655100 13952 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41507 } attempt: 4
I20260928 05:20:10.656015 13955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:10.655617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41507 (local address 127.0.0.1:47700)
0928 05:20:10.655808 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:10.655884 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260928 05:20:10.656921 13955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:10.656577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41507 (local address 127.0.0.1:47704)
0928 05:20:10.656715 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:10.656781 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260928 05:20:10.657115 13952 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41507: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41507: connect: Connection refused (error 111)
I20260928 05:20:10.659677 13959 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 05:20:10.662832 13974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:20:10.664090 13959 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 05:20:10.664242 13959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572622576160-388-0/minicluster-data/master-2/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572622576160-388-0/minicluster-data/master-2/wal
uuid: "a404cc2bb72145ffbad2de42faa3dafc"
format_stamp: "Formatted at 2026-09-28 05:20:10 on dist-test-slave-whsn"
server_key: "ac8b72635eb13810cf11830f06a23e5d"
server_key_iv: "626052cc7805d5bc8bc33eb073d8e734"
server_key_version: "encryptionkey@0"
I20260928 05:20:10.664413 13959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572622576160-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572622576160-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572622576160-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:20:10.687281 13959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:20:10.691001 13959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:20:10.691242 13959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:20:10.706485 13959 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:41507
I20260928 05:20:10.706518 14025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:41507 every 8 connection(s)
I20260928 05:20:10.707698 13959 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572622576160-388-0/minicluster-data/master-2/data/info.pb
I20260928 05:20:10.711265 14026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:20:10.711947 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 13959
I20260928 05:20:10.712111 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572622576160-388-0/minicluster-data/master-2/wal/instance
I20260928 05:20:10.712637 388 external_mini_cluster.cc:1428] Setting key 86a15849749b123ae53ba9252c881477
I20260928 05:20:10.720104 14026 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35731 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 05:20:10 dist-test-slave-whsn krb5kdc[13798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572809, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 28 05:20:10 dist-test-slave-whsn krb5kdc[13798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572810, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:20:10.733443 14029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:10.714367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47718 (local address 127.0.97.60:41507)
0928 05:20:10.715130 (+ 763us) server_negotiation.cc:206] Beginning negotiation
0928 05:20:10.715145 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 05:20:10.715185 (+ 40us) server_negotiation.cc:407] Connection header received
0928 05:20:10.716655 (+ 1470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:20:10.716675 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:20:10.717093 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:20:10.717436 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:20:10.718774 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:10.720055 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:10.721386 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:10.722014 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:10.722176 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:10.726230 (+ 4054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:20:10.726239 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:20:10.726252 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:20:10.726295 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:20:10.729841 (+ 3546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:10.730492 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:10.730499 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:10.730504 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:10.730652 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:10.730993 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:10.730997 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:10.730999 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:10.731436 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:20:10.731690 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0928 05:20:10.732109 (+ 419us) server_negotiation.cc:299] Negotiation successful
0928 05:20:10.732482 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":354,"threads_started":1}
I20260928 05:20:10.735819 13880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:10.722288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45596 (local address 127.0.97.62:35731)
0928 05:20:10.722503 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0928 05:20:10.722509 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:20:10.722989 (+ 480us) server_negotiation.cc:407] Connection header received
0928 05:20:10.723174 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:20:10.723178 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:20:10.723240 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:20:10.723321 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:20:10.724310 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:10.725217 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:10.726706 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:10.727226 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:10.727317 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:10.730848 (+ 3531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:20:10.730853 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:20:10.730856 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:20:10.730886 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:20:10.733271 (+ 2385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:10.734025 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:10.734029 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:10.734031 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:10.734151 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:10.734520 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:10.734524 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:10.734528 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:10.734753 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:20:10.734864 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 05:20:10.735353 (+ 489us) server_negotiation.cc:299] Negotiation successful
0928 05:20:10.735515 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 05:20:10.735940 14030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:10.722230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35731 (local address 127.0.0.1:45596)
0928 05:20:10.722846 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:10.722887 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 05:20:10.723044 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:20:10.723392 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:20:10.723398 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:20:10.723927 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:20:10.724176 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:20:10.724185 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:20:10.725377 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:20:10.725383 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:20:10.726572 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:20:10.726589 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:20:10.727403 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:20:10.727410 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:20:10.727590 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:10.728242 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:20:10.728263 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:20:10.730670 (+ 2407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:20:10.733397 (+ 2727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:20:10.733407 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:20:10.733414 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:20:10.733903 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:20:10.734251 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:20:10.734256 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:20:10.734259 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:20:10.734400 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:20:10.734865 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:20:10.734873 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:20:10.735074 (+ 201us) client_negotiation.cc:769] Sending connection context
0928 05:20:10.735270 (+ 196us) client_negotiation.cc:240] Negotiation successful
0928 05:20:10.735531 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":279,"threads_started":1}
I20260928 05:20:10.737206 14026 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44427 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 05:20:10 dist-test-slave-whsn krb5kdc[13798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572810, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Sep 28 05:20:10 dist-test-slave-whsn krb5kdc[13798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572809, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:20:10.748826 13878 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41507 } attempt: 3
I20260928 05:20:10.749418 14030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:10.737869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44427 (local address 127.0.0.1:33520)
0928 05:20:10.738005 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:10.738022 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 05:20:10.738099 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:20:10.738358 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:20:10.738360 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:20:10.738740 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:20:10.738897 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:20:10.738907 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:20:10.739914 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:20:10.739918 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:20:10.740779 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:20:10.740796 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:20:10.741517 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:20:10.741521 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:20:10.741663 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:10.742181 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:20:10.742196 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:20:10.744224 (+ 2028us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:20:10.747163 (+ 2939us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:20:10.747167 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:20:10.747170 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:20:10.747632 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:20:10.747967 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:20:10.747973 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:20:10.747977 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:20:10.748113 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:20:10.748567 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:20:10.748572 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:20:10.748702 (+ 130us) client_negotiation.cc:769] Sending connection context
0928 05:20:10.748823 (+ 121us) client_negotiation.cc:240] Negotiation successful
0928 05:20:10.749108 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 05:20:10.749816 13957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:10.737914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33520 (local address 127.0.97.61:44427)
0928 05:20:10.738101 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 05:20:10.738107 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:20:10.738126 (+ 19us) server_negotiation.cc:407] Connection header received
0928 05:20:10.738179 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:20:10.738183 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:20:10.738253 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:20:10.738353 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:20:10.739029 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:10.739774 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:10.740936 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:10.741377 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:10.741487 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:10.744403 (+ 2916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:20:10.744409 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:20:10.744412 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:20:10.744453 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:20:10.747042 (+ 2589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:10.747728 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:10.747731 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:10.747734 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:10.747856 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:10.748209 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:10.748214 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:10.748216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:10.748462 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:20:10.748583 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 05:20:10.748955 (+ 372us) server_negotiation.cc:299] Negotiation successful
0928 05:20:10.749221 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 05:20:10.750952 14026 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41507 } has no permanent_uuid. Determining permanent_uuid...
I20260928 05:20:10.751041 13880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:10.736557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45612 (local address 127.0.97.62:35731)
0928 05:20:10.736677 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 05:20:10.736682 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:20:10.736902 (+ 220us) server_negotiation.cc:407] Connection header received
0928 05:20:10.737031 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:20:10.737036 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:20:10.737103 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:20:10.737179 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:20:10.738491 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:10.739403 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:10.740716 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:10.741215 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:10.741307 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:10.745221 (+ 3914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:20:10.745225 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:20:10.745228 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:20:10.745256 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:20:10.747842 (+ 2586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:10.748411 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:10.748416 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:10.748420 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:10.748548 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:10.748924 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:10.748929 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:10.748932 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:10.749250 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:20:10.749520 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0928 05:20:10.750519 (+ 999us) server_negotiation.cc:299] Negotiation successful
0928 05:20:10.750730 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
Sep 28 05:20:10 dist-test-slave-whsn krb5kdc[13798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572809, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 28 05:20:10 dist-test-slave-whsn krb5kdc[13798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572810, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 28 05:20:10 dist-test-slave-whsn krb5kdc[13798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572809, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260928 05:20:10.765733 14029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:10.751379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47722 (local address 127.0.97.60:41507)
0928 05:20:10.751755 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0928 05:20:10.751760 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:20:10.751777 (+ 17us) server_negotiation.cc:407] Connection header received
0928 05:20:10.751819 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:20:10.751823 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:20:10.751885 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:20:10.751962 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:20:10.752823 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:10.753912 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:10.755576 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:10.756232 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:10.756338 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:10.759760 (+ 3422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:20:10.759766 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:20:10.759771 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:20:10.759799 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:20:10.762978 (+ 3179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:10.763661 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:10.763667 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:10.763670 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:10.763793 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:10.764185 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:10.764190 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:10.764193 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:10.764589 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:20:10.764730 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 05:20:10.765184 (+ 454us) server_negotiation.cc:299] Negotiation successful
0928 05:20:10.765395 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273}
I20260928 05:20:10.765750 13881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:10.749340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41507 (local address 127.0.0.1:47722)
0928 05:20:10.751052 (+ 1712us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:10.751071 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 05:20:10.751170 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:20:10.752008 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:20:10.752014 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:20:10.752481 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:20:10.752657 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:20:10.752669 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:20:10.754083 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:20:10.754089 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:20:10.755454 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:20:10.755468 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:20:10.756519 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:20:10.756525 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:20:10.756695 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:10.757211 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:20:10.757228 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:20:10.759610 (+ 2382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:20:10.763103 (+ 3493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:20:10.763108 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:20:10.763111 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:20:10.763543 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:20:10.763902 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:20:10.763907 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:20:10.763910 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:20:10.764057 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:20:10.764718 (+ 661us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:20:10.764723 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:20:10.764950 (+ 227us) client_negotiation.cc:769] Sending connection context
0928 05:20:10.765090 (+ 140us) client_negotiation.cc:240] Negotiation successful
0928 05:20:10.765350 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1476}
I20260928 05:20:10.766682 14030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:10.751660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41507 (local address 127.0.0.1:47734)
0928 05:20:10.752236 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:10.752251 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 05:20:10.752331 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:20:10.753788 (+ 1457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:20:10.753793 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:20:10.754200 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:20:10.754317 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:20:10.754325 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:20:10.755353 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:20:10.755358 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:20:10.756565 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:20:10.756577 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:20:10.757504 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:20:10.757511 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:20:10.757718 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:10.758238 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:20:10.758258 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:20:10.760735 (+ 2477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:20:10.764241 (+ 3506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:20:10.764246 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:20:10.764249 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:20:10.764615 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:20:10.765001 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:20:10.765006 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:20:10.765010 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:20:10.765155 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:20:10.765796 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:20:10.765800 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:20:10.765938 (+ 138us) client_negotiation.cc:769] Sending connection context
0928 05:20:10.766096 (+ 158us) client_negotiation.cc:240] Negotiation successful
0928 05:20:10.766369 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486}
I20260928 05:20:10.767445 14031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:10.752489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47734 (local address 127.0.97.60:41507)
0928 05:20:10.753415 (+ 926us) server_negotiation.cc:206] Beginning negotiation
0928 05:20:10.753430 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 05:20:10.753451 (+ 21us) server_negotiation.cc:407] Connection header received
0928 05:20:10.753492 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:20:10.753495 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:20:10.753541 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:20:10.753625 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:20:10.754475 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:10.755243 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:10.756699 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:10.757338 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:10.757468 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:10.761011 (+ 3543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:20:10.761018 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:20:10.761023 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:20:10.761057 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:20:10.764138 (+ 3081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:10.764722 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:10.764726 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:10.764729 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:10.764881 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:10.765275 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:10.765279 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:10.765283 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:10.765600 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:20:10.765738 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 05:20:10.766928 (+ 1190us) server_negotiation.cc:299] Negotiation successful
0928 05:20:10.767104 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":817,"thread_start_us":639,"threads_started":1}
I20260928 05:20:10.769308 13957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:10.753114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33528 (local address 127.0.97.61:44427)
0928 05:20:10.753344 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0928 05:20:10.753350 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:20:10.753615 (+ 265us) server_negotiation.cc:407] Connection header received
0928 05:20:10.753734 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:20:10.753739 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:20:10.753806 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:20:10.753889 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:20:10.755341 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:10.756109 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:10.757644 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:10.758109 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:10.758211 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:10.762682 (+ 4471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:20:10.762686 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:20:10.762689 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:20:10.762719 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:20:10.765612 (+ 2893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:10.767017 (+ 1405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:10.767028 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:10.767031 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:10.767189 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:10.767767 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:10.767772 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:10.767774 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:10.768068 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:20:10.768374 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0928 05:20:10.768800 (+ 426us) server_negotiation.cc:299] Negotiation successful
0928 05:20:10.769037 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
I20260928 05:20:10.771984 13878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f648f22b0d60437394cec07e21635a30: Bootstrap starting.
I20260928 05:20:10.772523 14026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a404cc2bb72145ffbad2de42faa3dafc: Bootstrap starting.
I20260928 05:20:10.775871 13878 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f648f22b0d60437394cec07e21635a30: Neither blocks nor log segments found. Creating new log.
I20260928 05:20:10.775872 14026 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a404cc2bb72145ffbad2de42faa3dafc: Neither blocks nor log segments found. Creating new log.
I20260928 05:20:10.777184 13878 log.cc:824] T 00000000000000000000000000000000 P f648f22b0d60437394cec07e21635a30: Log is configured to *not* fsync() on all Append() calls
I20260928 05:20:10.777942 14026 log.cc:824] T 00000000000000000000000000000000 P a404cc2bb72145ffbad2de42faa3dafc: Log is configured to *not* fsync() on all Append() calls
I20260928 05:20:10.780212 13878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f648f22b0d60437394cec07e21635a30: No bootstrap required, opened a new log
I20260928 05:20:10.780397 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 05:20:10.780766 14026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a404cc2bb72145ffbad2de42faa3dafc: No bootstrap required, opened a new log
I20260928 05:20:10.788153 13878 raft_consensus.cc:348] T 00000000000000000000000000000000 P f648f22b0d60437394cec07e21635a30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f648f22b0d60437394cec07e21635a30" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35731 } } peers { permanent_uuid: "b5005445bae743bebdde0d6021142b9c" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44427 } } peers { permanent_uuid: "a404cc2bb72145ffbad2de42faa3dafc" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41507 } }
I20260928 05:20:10.788552 13878 raft_consensus.cc:374] T 00000000000000000000000000000000 P f648f22b0d60437394cec07e21635a30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:20:10.788651 13878 raft_consensus.cc:729] T 00000000000000000000000000000000 P f648f22b0d60437394cec07e21635a30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f648f22b0d60437394cec07e21635a30, State: Initialized, Role: FOLLOWER
I20260928 05:20:10.789405 13878 consensus_queue.cc:260] T 00000000000000000000000000000000 P f648f22b0d60437394cec07e21635a30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f648f22b0d60437394cec07e21635a30" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35731 } } peers { permanent_uuid: "b5005445bae743bebdde0d6021142b9c" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44427 } } peers { permanent_uuid: "a404cc2bb72145ffbad2de42faa3dafc" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41507 } }
I20260928 05:20:10.791286 14026 raft_consensus.cc:348] T 00000000000000000000000000000000 P a404cc2bb72145ffbad2de42faa3dafc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f648f22b0d60437394cec07e21635a30" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35731 } } peers { permanent_uuid: "b5005445bae743bebdde0d6021142b9c" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44427 } } peers { permanent_uuid: "a404cc2bb72145ffbad2de42faa3dafc" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41507 } }
I20260928 05:20:10.791689 14026 raft_consensus.cc:374] T 00000000000000000000000000000000 P a404cc2bb72145ffbad2de42faa3dafc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:20:10.791800 14026 raft_consensus.cc:729] T 00000000000000000000000000000000 P a404cc2bb72145ffbad2de42faa3dafc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a404cc2bb72145ffbad2de42faa3dafc, State: Initialized, Role: FOLLOWER
I20260928 05:20:10.792411 14026 consensus_queue.cc:260] T 00000000000000000000000000000000 P a404cc2bb72145ffbad2de42faa3dafc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f648f22b0d60437394cec07e21635a30" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35731 } } peers { permanent_uuid: "b5005445bae743bebdde0d6021142b9c" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44427 } } peers { permanent_uuid: "a404cc2bb72145ffbad2de42faa3dafc" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41507 } }
I20260928 05:20:10.793987 14040 sys_catalog.cc:455] T 00000000000000000000000000000000 P f648f22b0d60437394cec07e21635a30 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f648f22b0d60437394cec07e21635a30" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35731 } } peers { permanent_uuid: "b5005445bae743bebdde0d6021142b9c" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44427 } } peers { permanent_uuid: "a404cc2bb72145ffbad2de42faa3dafc" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41507 } } }
I20260928 05:20:10.794363 14040 sys_catalog.cc:458] T 00000000000000000000000000000000 P f648f22b0d60437394cec07e21635a30 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 05:20:10.796008 13878 sys_catalog.cc:564] T 00000000000000000000000000000000 P f648f22b0d60437394cec07e21635a30 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:20:10.802704 14041 sys_catalog.cc:455] T 00000000000000000000000000000000 P a404cc2bb72145ffbad2de42faa3dafc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f648f22b0d60437394cec07e21635a30" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35731 } } peers { permanent_uuid: "b5005445bae743bebdde0d6021142b9c" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44427 } } peers { permanent_uuid: "a404cc2bb72145ffbad2de42faa3dafc" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41507 } } }
I20260928 05:20:10.803206 14041 sys_catalog.cc:458] T 00000000000000000000000000000000 P a404cc2bb72145ffbad2de42faa3dafc [sys.catalog]: This master's current role is: FOLLOWER
I20260928 05:20:10.810917 13957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:10.787270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33540 (local address 127.0.97.61:44427)
0928 05:20:10.787443 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 05:20:10.787449 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:20:10.790713 (+ 3264us) server_negotiation.cc:407] Connection header received
0928 05:20:10.791454 (+ 741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:20:10.791461 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:20:10.791548 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:20:10.791641 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:20:10.794346 (+ 2705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:10.795375 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:10.797678 (+ 2303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:10.798347 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:10.798439 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:10.803503 (+ 5064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:20:10.803510 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:20:10.803516 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:20:10.803561 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:20:10.807162 (+ 3601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:10.808246 (+ 1084us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:10.808253 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:10.808257 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:10.808420 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:10.809032 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:10.809037 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:10.809041 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:10.809448 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:20:10.809673 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0928 05:20:10.810279 (+ 606us) server_negotiation.cc:299] Negotiation successful
0928 05:20:10.810538 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 05:20:10.824872 14026 sys_catalog.cc:564] T 00000000000000000000000000000000 P a404cc2bb72145ffbad2de42faa3dafc [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:20:10.829618 13880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:10.787565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45616 (local address 127.0.97.62:35731)
0928 05:20:10.787711 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 05:20:10.787716 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:20:10.812300 (+ 24584us) server_negotiation.cc:407] Connection header received
0928 05:20:10.812575 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:20:10.812582 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:20:10.812666 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:20:10.812738 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:20:10.814700 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:10.815877 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:10.818036 (+ 2159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:10.818729 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:10.818822 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:10.822633 (+ 3811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:20:10.822641 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:20:10.822645 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:20:10.822686 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:20:10.826421 (+ 3735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:10.827416 (+ 995us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:10.827431 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:10.827435 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:10.827630 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:10.828059 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:10.828065 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:10.828069 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:10.828396 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:20:10.828663 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0928 05:20:10.829013 (+ 350us) server_negotiation.cc:299] Negotiation successful
0928 05:20:10.829244 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 05:20:10.839352 14031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:10.793906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47748 (local address 127.0.97.60:41507)
0928 05:20:10.794060 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 05:20:10.794066 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:20:10.816109 (+ 22043us) server_negotiation.cc:407] Connection header received
0928 05:20:10.816591 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:20:10.816597 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:20:10.816694 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:20:10.816764 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:20:10.826110 (+ 9346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:10.826962 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:10.828645 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:10.829182 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:10.829271 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:10.832457 (+ 3186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:20:10.832464 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:20:10.832468 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:20:10.832507 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:20:10.835970 (+ 3463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:10.836905 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:10.836910 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:10.836914 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:10.837066 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:10.837520 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:10.837526 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:10.837530 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:10.837906 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:20:10.838151 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0928 05:20:10.838733 (+ 582us) server_negotiation.cc:299] Negotiation successful
0928 05:20:10.838986 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260928 05:20:10.847326 14055 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f648f22b0d60437394cec07e21635a30: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 05:20:10.847527 14055 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 05:20:10.835964 13811 master_runner.cc:186] Error getting master registration for 127.0.97.61:44427: OK, instance_id { permanent_uuid: "b5005445bae743bebdde0d6021142b9c" instance_seqno: 1790572810236098 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 05:20:10.847888 13811 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260928 05:20:10.861150 14065 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a404cc2bb72145ffbad2de42faa3dafc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 05:20:10.861392 14065 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 05:20:10.861794 13959 master_runner.cc:186] Error getting master registration for 127.0.97.61:44427: OK, instance_id { permanent_uuid: "b5005445bae743bebdde0d6021142b9c" instance_seqno: 1790572810236098 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 05:20:10.871295 13959 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260928 05:20:10.943161 13980 tablet.cc:2369] T 00000000000000000000000000000000 P a404cc2bb72145ffbad2de42faa3dafc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 05:20:10.956313 13952 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41507 } attempt: 5
Sep 28 05:20:10 dist-test-slave-whsn krb5kdc[13798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572810, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260928 05:20:10.975303 13955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:10.956912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41507 (local address 127.0.0.1:47752)
0928 05:20:10.957171 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:10.957190 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 05:20:10.957293 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:20:10.957634 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:20:10.957639 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:20:10.958152 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:20:10.958358 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:20:10.958370 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:20:10.959843 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:20:10.959850 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:20:10.961320 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:20:10.961335 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:20:10.962460 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:20:10.962467 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:20:10.962669 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:10.963337 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:20:10.963360 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:20:10.966138 (+ 2778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:20:10.972438 (+ 6300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:20:10.972444 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:20:10.972448 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:20:10.972986 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:20:10.973422 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:20:10.973427 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:20:10.973430 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:20:10.973597 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:20:10.974298 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:20:10.974303 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:20:10.974471 (+ 168us) client_negotiation.cc:769] Sending connection context
0928 05:20:10.974613 (+ 142us) client_negotiation.cc:240] Negotiation successful
0928 05:20:10.974900 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140}
I20260928 05:20:10.975437 14031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:10.957012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47752 (local address 127.0.97.60:41507)
0928 05:20:10.957218 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 05:20:10.957225 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:20:10.957248 (+ 23us) server_negotiation.cc:407] Connection header received
0928 05:20:10.957411 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:20:10.957417 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:20:10.957494 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:20:10.957606 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:20:10.958522 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:10.959685 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:10.961492 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:10.962313 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:10.962452 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:10.966339 (+ 3887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:20:10.966346 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:20:10.966351 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:20:10.966404 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:20:10.972268 (+ 5864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:10.973125 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:10.973132 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:10.973137 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:10.973304 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:10.973720 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:10.973726 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:10.973730 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:10.974171 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:20:10.974352 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0928 05:20:10.974774 (+ 422us) server_negotiation.cc:299] Negotiation successful
0928 05:20:10.975024 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260928 05:20:10.981691 13952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5005445bae743bebdde0d6021142b9c: Bootstrap starting.
I20260928 05:20:10.985852 13952 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5005445bae743bebdde0d6021142b9c: Neither blocks nor log segments found. Creating new log.
I20260928 05:20:10.987205 13952 log.cc:824] T 00000000000000000000000000000000 P b5005445bae743bebdde0d6021142b9c: Log is configured to *not* fsync() on all Append() calls
I20260928 05:20:10.992591 13952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5005445bae743bebdde0d6021142b9c: No bootstrap required, opened a new log
I20260928 05:20:11.003434 13952 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5005445bae743bebdde0d6021142b9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f648f22b0d60437394cec07e21635a30" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35731 } } peers { permanent_uuid: "b5005445bae743bebdde0d6021142b9c" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44427 } } peers { permanent_uuid: "a404cc2bb72145ffbad2de42faa3dafc" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41507 } }
I20260928 05:20:11.003881 13952 raft_consensus.cc:374] T 00000000000000000000000000000000 P b5005445bae743bebdde0d6021142b9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:20:11.004002 13952 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5005445bae743bebdde0d6021142b9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5005445bae743bebdde0d6021142b9c, State: Initialized, Role: FOLLOWER
I20260928 05:20:11.004680 13952 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5005445bae743bebdde0d6021142b9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f648f22b0d60437394cec07e21635a30" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35731 } } peers { permanent_uuid: "b5005445bae743bebdde0d6021142b9c" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44427 } } peers { permanent_uuid: "a404cc2bb72145ffbad2de42faa3dafc" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41507 } }
I20260928 05:20:11.007046 13952 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5005445bae743bebdde0d6021142b9c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:20:11.005697 14068 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5005445bae743bebdde0d6021142b9c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f648f22b0d60437394cec07e21635a30" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35731 } } peers { permanent_uuid: "b5005445bae743bebdde0d6021142b9c" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44427 } } peers { permanent_uuid: "a404cc2bb72145ffbad2de42faa3dafc" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41507 } } }
I20260928 05:20:11.011404 14068 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5005445bae743bebdde0d6021142b9c [sys.catalog]: This master's current role is: FOLLOWER
I20260928 05:20:11.024338 13885 master_runner.cc:263] No leader master found from master b5005445bae743bebdde0d6021142b9c
I20260928 05:20:11.024497 13885 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260928 05:20:11.026962 14079 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b5005445bae743bebdde0d6021142b9c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 05:20:11.027151 14079 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260928 05:20:11.036671 13832 tablet.cc:2369] T 00000000000000000000000000000000 P f648f22b0d60437394cec07e21635a30: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 05:20:11.100729 14040 raft_consensus.cc:482] T 00000000000000000000000000000000 P f648f22b0d60437394cec07e21635a30 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:20:11.101110 14040 raft_consensus.cc:504] T 00000000000000000000000000000000 P f648f22b0d60437394cec07e21635a30 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f648f22b0d60437394cec07e21635a30" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35731 } } peers { permanent_uuid: "b5005445bae743bebdde0d6021142b9c" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44427 } } peers { permanent_uuid: "a404cc2bb72145ffbad2de42faa3dafc" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41507 } }
I20260928 05:20:11.103796 14040 leader_election.cc:288] T 00000000000000000000000000000000 P f648f22b0d60437394cec07e21635a30 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5005445bae743bebdde0d6021142b9c (127.0.97.61:44427), a404cc2bb72145ffbad2de42faa3dafc (127.0.97.60:41507)
I20260928 05:20:11.104439 13927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f648f22b0d60437394cec07e21635a30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5005445bae743bebdde0d6021142b9c" is_pre_election: true
I20260928 05:20:11.104771 14001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f648f22b0d60437394cec07e21635a30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a404cc2bb72145ffbad2de42faa3dafc" is_pre_election: true
I20260928 05:20:11.105147 13927 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b5005445bae743bebdde0d6021142b9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f648f22b0d60437394cec07e21635a30 in term 0.
I20260928 05:20:11.105387 14001 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a404cc2bb72145ffbad2de42faa3dafc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f648f22b0d60437394cec07e21635a30 in term 0.
I20260928 05:20:11.105924 13827 leader_election.cc:302] T 00000000000000000000000000000000 P f648f22b0d60437394cec07e21635a30 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b5005445bae743bebdde0d6021142b9c, f648f22b0d60437394cec07e21635a30; no voters:
I20260928 05:20:11.107784 14040 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f648f22b0d60437394cec07e21635a30 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 05:20:11.107941 14040 raft_consensus.cc:482] T 00000000000000000000000000000000 P f648f22b0d60437394cec07e21635a30 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 05:20:11.108049 14040 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f648f22b0d60437394cec07e21635a30 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:20:11.111426 14040 raft_consensus.cc:504] T 00000000000000000000000000000000 P f648f22b0d60437394cec07e21635a30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f648f22b0d60437394cec07e21635a30" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35731 } } peers { permanent_uuid: "b5005445bae743bebdde0d6021142b9c" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44427 } } peers { permanent_uuid: "a404cc2bb72145ffbad2de42faa3dafc" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41507 } }
I20260928 05:20:11.112572 13927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f648f22b0d60437394cec07e21635a30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5005445bae743bebdde0d6021142b9c"
I20260928 05:20:11.112794 13927 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5005445bae743bebdde0d6021142b9c [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:20:11.116205 13927 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b5005445bae743bebdde0d6021142b9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f648f22b0d60437394cec07e21635a30 in term 1.
I20260928 05:20:11.116675 13827 leader_election.cc:302] T 00000000000000000000000000000000 P f648f22b0d60437394cec07e21635a30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b5005445bae743bebdde0d6021142b9c, f648f22b0d60437394cec07e21635a30; no voters:
I20260928 05:20:11.119972 14001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f648f22b0d60437394cec07e21635a30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a404cc2bb72145ffbad2de42faa3dafc"
I20260928 05:20:11.120229 14001 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a404cc2bb72145ffbad2de42faa3dafc [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:20:11.123683 14001 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a404cc2bb72145ffbad2de42faa3dafc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f648f22b0d60437394cec07e21635a30 in term 1.
I20260928 05:20:11.124400 14040 leader_election.cc:288] T 00000000000000000000000000000000 P f648f22b0d60437394cec07e21635a30 [CANDIDATE]: Term 1 election: Requested vote from peers b5005445bae743bebdde0d6021142b9c (127.0.97.61:44427), a404cc2bb72145ffbad2de42faa3dafc (127.0.97.60:41507)
I20260928 05:20:11.124639 14040 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f648f22b0d60437394cec07e21635a30 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:20:11.124794 14040 raft_consensus.cc:686] T 00000000000000000000000000000000 P f648f22b0d60437394cec07e21635a30 [term 1 LEADER]: Becoming Leader. State: Replica: f648f22b0d60437394cec07e21635a30, State: Running, Role: LEADER
I20260928 05:20:11.126340 14040 consensus_queue.cc:237] T 00000000000000000000000000000000 P f648f22b0d60437394cec07e21635a30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f648f22b0d60437394cec07e21635a30" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35731 } } peers { permanent_uuid: "b5005445bae743bebdde0d6021142b9c" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44427 } } peers { permanent_uuid: "a404cc2bb72145ffbad2de42faa3dafc" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41507 } }
I20260928 05:20:11.144758 14081 sys_catalog.cc:455] T 00000000000000000000000000000000 P f648f22b0d60437394cec07e21635a30 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f648f22b0d60437394cec07e21635a30. Latest consensus state: current_term: 1 leader_uuid: "f648f22b0d60437394cec07e21635a30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f648f22b0d60437394cec07e21635a30" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35731 } } peers { permanent_uuid: "b5005445bae743bebdde0d6021142b9c" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44427 } } peers { permanent_uuid: "a404cc2bb72145ffbad2de42faa3dafc" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41507 } } }
I20260928 05:20:11.145215 14081 sys_catalog.cc:458] T 00000000000000000000000000000000 P f648f22b0d60437394cec07e21635a30 [sys.catalog]: This master's current role is: LEADER
I20260928 05:20:11.153890 14083 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:20:11.157982 14083 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:20:11.186259 14001 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a404cc2bb72145ffbad2de42faa3dafc [term 1 FOLLOWER]: Refusing update from remote peer f648f22b0d60437394cec07e21635a30: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:20:11.187382 14081 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f648f22b0d60437394cec07e21635a30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a404cc2bb72145ffbad2de42faa3dafc" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:20:11.196372 13927 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b5005445bae743bebdde0d6021142b9c [term 1 FOLLOWER]: Refusing update from remote peer f648f22b0d60437394cec07e21635a30: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:20:11.197456 14081 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f648f22b0d60437394cec07e21635a30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5005445bae743bebdde0d6021142b9c" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 05:20:11.226708 13906 tablet.cc:2369] T 00000000000000000000000000000000 P b5005445bae743bebdde0d6021142b9c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 05:20:11.229991 14068 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5005445bae743bebdde0d6021142b9c [sys.catalog]: SysCatalogTable state changed. Reason: New leader f648f22b0d60437394cec07e21635a30. Latest consensus state: current_term: 1 leader_uuid: "f648f22b0d60437394cec07e21635a30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f648f22b0d60437394cec07e21635a30" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35731 } } peers { permanent_uuid: "b5005445bae743bebdde0d6021142b9c" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44427 } } peers { permanent_uuid: "a404cc2bb72145ffbad2de42faa3dafc" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41507 } } }
I20260928 05:20:11.230306 14068 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5005445bae743bebdde0d6021142b9c [sys.catalog]: This master's current role is: FOLLOWER
I20260928 05:20:11.235195 14084 mvcc.cc:205] Tried to move back new op lower bound from 7334186234606350336 to 7334186234393075712. Current Snapshot: MvccSnapshot[applied={T|T < 7334186234606350336}]
I20260928 05:20:11.239935 14086 mvcc.cc:205] Tried to move back new op lower bound from 7334186234606350336 to 7334186234393075712. Current Snapshot: MvccSnapshot[applied={T|T < 7334186234606350336}]
I20260928 05:20:11.261631 14068 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5005445bae743bebdde0d6021142b9c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f648f22b0d60437394cec07e21635a30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f648f22b0d60437394cec07e21635a30" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35731 } } peers { permanent_uuid: "b5005445bae743bebdde0d6021142b9c" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44427 } } peers { permanent_uuid: "a404cc2bb72145ffbad2de42faa3dafc" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41507 } } }
I20260928 05:20:11.261911 14068 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5005445bae743bebdde0d6021142b9c [sys.catalog]: This master's current role is: FOLLOWER
I20260928 05:20:11.270437 14041 sys_catalog.cc:455] T 00000000000000000000000000000000 P a404cc2bb72145ffbad2de42faa3dafc [sys.catalog]: SysCatalogTable state changed. Reason: New leader f648f22b0d60437394cec07e21635a30. Latest consensus state: current_term: 1 leader_uuid: "f648f22b0d60437394cec07e21635a30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f648f22b0d60437394cec07e21635a30" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35731 } } peers { permanent_uuid: "b5005445bae743bebdde0d6021142b9c" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44427 } } peers { permanent_uuid: "a404cc2bb72145ffbad2de42faa3dafc" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41507 } } }
I20260928 05:20:11.270742 14041 sys_catalog.cc:458] T 00000000000000000000000000000000 P a404cc2bb72145ffbad2de42faa3dafc [sys.catalog]: This master's current role is: FOLLOWER
I20260928 05:20:11.273954 14085 mvcc.cc:205] Tried to move back new op lower bound from 7334186234606350336 to 7334186234393075712. Current Snapshot: MvccSnapshot[applied={T|T < 7334186234606350336}]
I20260928 05:20:11.277284 14081 sys_catalog.cc:455] T 00000000000000000000000000000000 P f648f22b0d60437394cec07e21635a30 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f648f22b0d60437394cec07e21635a30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f648f22b0d60437394cec07e21635a30" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35731 } } peers { permanent_uuid: "b5005445bae743bebdde0d6021142b9c" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44427 } } peers { permanent_uuid: "a404cc2bb72145ffbad2de42faa3dafc" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41507 } } }
I20260928 05:20:11.277549 14081 sys_catalog.cc:458] T 00000000000000000000000000000000 P f648f22b0d60437394cec07e21635a30 [sys.catalog]: This master's current role is: LEADER
I20260928 05:20:11.277858 14081 sys_catalog.cc:455] T 00000000000000000000000000000000 P f648f22b0d60437394cec07e21635a30 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f648f22b0d60437394cec07e21635a30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f648f22b0d60437394cec07e21635a30" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35731 } } peers { permanent_uuid: "b5005445bae743bebdde0d6021142b9c" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44427 } } peers { permanent_uuid: "a404cc2bb72145ffbad2de42faa3dafc" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41507 } } }
I20260928 05:20:11.278055 14081 sys_catalog.cc:458] T 00000000000000000000000000000000 P f648f22b0d60437394cec07e21635a30 [sys.catalog]: This master's current role is: LEADER
I20260928 05:20:11.279265 14083 catalog_manager.cc:1347] Generated new cluster ID: 6b302b470645487cabf99d345eb95bdf
I20260928 05:20:11.279361 14083 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:20:11.283080 14041 sys_catalog.cc:455] T 00000000000000000000000000000000 P a404cc2bb72145ffbad2de42faa3dafc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f648f22b0d60437394cec07e21635a30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f648f22b0d60437394cec07e21635a30" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35731 } } peers { permanent_uuid: "b5005445bae743bebdde0d6021142b9c" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44427 } } peers { permanent_uuid: "a404cc2bb72145ffbad2de42faa3dafc" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41507 } } }
I20260928 05:20:11.283324 14041 sys_catalog.cc:458] T 00000000000000000000000000000000 P a404cc2bb72145ffbad2de42faa3dafc [sys.catalog]: This master's current role is: FOLLOWER
I20260928 05:20:11.312395 14083 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:20:11.314229 14083 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:20:11.330224 14083 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f648f22b0d60437394cec07e21635a30: Generated new TSK 0
I20260928 05:20:11.331282 14083 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 05:20:11.863298 14065 catalog_manager.cc:1259] Loaded cluster ID: 6b302b470645487cabf99d345eb95bdf
I20260928 05:20:11.863397 14065 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a404cc2bb72145ffbad2de42faa3dafc: loading cluster ID for follower catalog manager: success
I20260928 05:20:11.865710 14065 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a404cc2bb72145ffbad2de42faa3dafc: acquiring CA information for follower catalog manager: success
I20260928 05:20:11.867368 14065 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a404cc2bb72145ffbad2de42faa3dafc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 05:20:12.029039 14079 catalog_manager.cc:1259] Loaded cluster ID: 6b302b470645487cabf99d345eb95bdf
I20260928 05:20:12.029136 14079 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b5005445bae743bebdde0d6021142b9c: loading cluster ID for follower catalog manager: success
I20260928 05:20:12.031502 14079 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b5005445bae743bebdde0d6021142b9c: acquiring CA information for follower catalog manager: success
I20260928 05:20:12.033169 14079 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b5005445bae743bebdde0d6021142b9c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 05:20:12.426714 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 13811
I20260928 05:20:12.427929 13873 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:20:12.540401 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 13811
I20260928 05:20:12.556777 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 13885
I20260928 05:20:12.557756 13947 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:20:12.659646 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 13885
I20260928 05:20:12.675661 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 13959
I20260928 05:20:12.676661 14021 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:20:12.780624 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 13959
2026-09-28T05:20:12Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4580 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790572622576160-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:20:12 dist-test-slave-whsn krb5kdc[14101](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:20:12 dist-test-slave-whsn krb5kdc[14101](info): set up 2 sockets
Sep 28 05:20:12 dist-test-slave-whsn krb5kdc[14101](info): commencing operation
krb5kdc: starting...
W20260928 05:20:13.776885 388 mini_kdc.cc:121] Time spent starting KDC: real 0.960s user 0.002s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790572622576160-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790572622576160-388-0/krb5kdc/test-user.keytab.
Sep 28 05:20:13 dist-test-slave-whsn krb5kdc[14101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572813, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:20:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:20:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 05:20:13.971629 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790572622576160-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790572622576160-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790572622576160-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46747
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46747
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:20:14.270268 14114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:20:14.270557 14114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:20:14.270699 14114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:20:14.277659 14114 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:20:14.277760 14114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:20:14.277835 14114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:20:14.277908 14114 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:20:14.277969 14114 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:20:14.287223 14114 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790572622576160-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790572622576160-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46747
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46747
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790572622576160-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:20:14.288381 14114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:20:14.289647 14114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:20:14.300102 14122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:20:14.300148 14119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:20:14.300145 14120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:20:14.301056 14114 server_base.cc:1034] running on GCE node
I20260928 05:20:14.301671 14114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:20:14.303447 14114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:20:14.304636 14114 hybrid_clock.cc:648] HybridClock initialized: now 1790572814304581 us; error 45 us; skew 500 ppm
I20260928 05:20:14.304972 14114 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:20:14 dist-test-slave-whsn krb5kdc[14101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572814, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:20:14.307754 14114 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 05:20:14.309090 14114 webserver.cc:460] Webserver started at http://127.0.97.62:42377/ using document root <none> and password file <none>
I20260928 05:20:14.309692 14114 fs_manager.cc:359] Metadata directory not provided
I20260928 05:20:14.309800 14114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:20:14.310081 14114 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:20:14.312798 14114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790572622576160-388-0/minicluster-data/master-0/data/instance:
uuid: "0d8bb262c7254a0c9d16e0d3a5eb3ecf"
format_stamp: "Formatted at 2026-09-28 05:20:14 on dist-test-slave-whsn"
server_key: "9f3f81fdfc988dae78f763864d71482d"
server_key_iv: "24705a1e858eea137aee0f638b794451"
server_key_version: "encryptionkey@0"
I20260928 05:20:14.313529 14114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790572622576160-388-0/minicluster-data/master-0/wal/instance:
uuid: "0d8bb262c7254a0c9d16e0d3a5eb3ecf"
format_stamp: "Formatted at 2026-09-28 05:20:14 on dist-test-slave-whsn"
server_key: "9f3f81fdfc988dae78f763864d71482d"
server_key_iv: "24705a1e858eea137aee0f638b794451"
server_key_version: "encryptionkey@0"
I20260928 05:20:14.318280 14114 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 05:20:14.321389 14129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:20:14.322558 14114 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 05:20:14.322701 14114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790572622576160-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790572622576160-388-0/minicluster-data/master-0/wal
uuid: "0d8bb262c7254a0c9d16e0d3a5eb3ecf"
format_stamp: "Formatted at 2026-09-28 05:20:14 on dist-test-slave-whsn"
server_key: "9f3f81fdfc988dae78f763864d71482d"
server_key_iv: "24705a1e858eea137aee0f638b794451"
server_key_version: "encryptionkey@0"
I20260928 05:20:14.322860 14114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790572622576160-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790572622576160-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790572622576160-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:20:14.351292 14114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:20:14.355224 14114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:20:14.355470 14114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:20:14.375525 14114 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46747
I20260928 05:20:14.375583 14180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46747 every 8 connection(s)
I20260928 05:20:14.377066 14114 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:20:14.377544 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 14114
I20260928 05:20:14.377780 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790572622576160-388-0/minicluster-data/master-0/wal/instance
I20260928 05:20:14.378495 388 external_mini_cluster.cc:1428] Setting key b515abd7d6b2a78452dd49ac675b6207
I20260928 05:20:14.381214 14181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 05:20:14 dist-test-slave-whsn krb5kdc[14101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572813, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:20:14.393397 14181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d8bb262c7254a0c9d16e0d3a5eb3ecf: Bootstrap starting.
I20260928 05:20:14.397244 14181 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d8bb262c7254a0c9d16e0d3a5eb3ecf: Neither blocks nor log segments found. Creating new log.
I20260928 05:20:14.398423 14181 log.cc:824] T 00000000000000000000000000000000 P 0d8bb262c7254a0c9d16e0d3a5eb3ecf: Log is configured to *not* fsync() on all Append() calls
I20260928 05:20:14.401131 14184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:14.380194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48756 (local address 127.0.97.62:46747)
0928 05:20:14.380964 (+ 770us) server_negotiation.cc:206] Beginning negotiation
0928 05:20:14.380976 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 05:20:14.381022 (+ 46us) server_negotiation.cc:407] Connection header received
0928 05:20:14.382636 (+ 1614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:20:14.382657 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:20:14.383019 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:20:14.383349 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:20:14.385049 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:14.386392 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:14.388124 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:14.388766 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:14.388926 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:14.393330 (+ 4404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:20:14.393341 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:20:14.393360 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:20:14.393406 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:20:14.397021 (+ 3615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:14.397653 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:14.397662 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:14.397670 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:14.397863 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:14.398187 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:14.398192 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:14.398195 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:14.398847 (+ 652us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:20:14.399141 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0928 05:20:14.399505 (+ 364us) server_negotiation.cc:299] Negotiation successful
0928 05:20:14.399871 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":524,"thread_start_us":318,"threads_started":1}
I20260928 05:20:14.401666 14181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d8bb262c7254a0c9d16e0d3a5eb3ecf: No bootstrap required, opened a new log
I20260928 05:20:14.404650 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 05:20:14.410002 14181 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d8bb262c7254a0c9d16e0d3a5eb3ecf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d8bb262c7254a0c9d16e0d3a5eb3ecf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46747 } }
I20260928 05:20:14.410333 14181 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d8bb262c7254a0c9d16e0d3a5eb3ecf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:20:14.410400 14181 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d8bb262c7254a0c9d16e0d3a5eb3ecf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d8bb262c7254a0c9d16e0d3a5eb3ecf, State: Initialized, Role: FOLLOWER
I20260928 05:20:14.410861 14181 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d8bb262c7254a0c9d16e0d3a5eb3ecf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d8bb262c7254a0c9d16e0d3a5eb3ecf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46747 } }
I20260928 05:20:14.411078 14181 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d8bb262c7254a0c9d16e0d3a5eb3ecf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:20:14.411171 14181 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d8bb262c7254a0c9d16e0d3a5eb3ecf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:20:14.411288 14181 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d8bb262c7254a0c9d16e0d3a5eb3ecf [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:20:14.413283 14181 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d8bb262c7254a0c9d16e0d3a5eb3ecf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d8bb262c7254a0c9d16e0d3a5eb3ecf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46747 } }
I20260928 05:20:14.413681 14181 leader_election.cc:302] T 00000000000000000000000000000000 P 0d8bb262c7254a0c9d16e0d3a5eb3ecf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d8bb262c7254a0c9d16e0d3a5eb3ecf; no voters:
I20260928 05:20:14.414144 14181 leader_election.cc:288] T 00000000000000000000000000000000 P 0d8bb262c7254a0c9d16e0d3a5eb3ecf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:20:14.414261 14186 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d8bb262c7254a0c9d16e0d3a5eb3ecf [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:20:14.414844 14186 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d8bb262c7254a0c9d16e0d3a5eb3ecf [term 1 LEADER]: Becoming Leader. State: Replica: 0d8bb262c7254a0c9d16e0d3a5eb3ecf, State: Running, Role: LEADER
I20260928 05:20:14.415302 14186 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d8bb262c7254a0c9d16e0d3a5eb3ecf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d8bb262c7254a0c9d16e0d3a5eb3ecf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46747 } }
I20260928 05:20:14.415922 14181 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d8bb262c7254a0c9d16e0d3a5eb3ecf [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:20:14.418023 14187 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d8bb262c7254a0c9d16e0d3a5eb3ecf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d8bb262c7254a0c9d16e0d3a5eb3ecf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d8bb262c7254a0c9d16e0d3a5eb3ecf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46747 } } }
I20260928 05:20:14.418399 14187 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d8bb262c7254a0c9d16e0d3a5eb3ecf [sys.catalog]: This master's current role is: LEADER
I20260928 05:20:14.418886 14188 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d8bb262c7254a0c9d16e0d3a5eb3ecf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d8bb262c7254a0c9d16e0d3a5eb3ecf. Latest consensus state: current_term: 1 leader_uuid: "0d8bb262c7254a0c9d16e0d3a5eb3ecf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d8bb262c7254a0c9d16e0d3a5eb3ecf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46747 } } }
I20260928 05:20:14.419168 14188 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d8bb262c7254a0c9d16e0d3a5eb3ecf [sys.catalog]: This master's current role is: LEADER
I20260928 05:20:14.421777 14194 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:20:14.425870 14194 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:20:14.433506 14194 catalog_manager.cc:1347] Generated new cluster ID: 5c8c87dfcff24c0a8b307139c61fb8e8
I20260928 05:20:14.433605 14194 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:20:14.460069 14194 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:20:14.461892 14194 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:20:14.471418 14194 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d8bb262c7254a0c9d16e0d3a5eb3ecf: Generated new TSK 0
I20260928 05:20:14.472150 14194 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 05:20:14.516760 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 14114
I20260928 05:20:14.517906 14176 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:20:14.610396 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 14114
2026-09-28T05:20:14Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1826 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:20:14 dist-test-slave-whsn krb5kdc[14209](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:20:14 dist-test-slave-whsn krb5kdc[14209](info): set up 2 sockets
Sep 28 05:20:14 dist-test-slave-whsn krb5kdc[14209](info): commencing operation
krb5kdc: starting...
W20260928 05:20:15.600998 388 mini_kdc.cc:121] Time spent starting KDC: real 0.958s user 0.001s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/krb5kdc/test-user.keytab.
Sep 28 05:20:15 dist-test-slave-whsn krb5kdc[14209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572815, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:20:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:20:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 05:20:15.789311 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38873
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38873
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:20:16.091230 14222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:20:16.091506 14222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:20:16.091650 14222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:20:16.098601 14222 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:20:16.098707 14222 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:20:16.098783 14222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:20:16.098860 14222 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:20:16.098919 14222 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:20:16.108174 14222 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38873
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:20:16.109345 14222 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:20:16.110539 14222 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:20:16.120966 14227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:20:16.121114 14228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:20:16.121546 14230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:20:16.121754 14222 server_base.cc:1034] running on GCE node
I20260928 05:20:16.122592 14222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:20:16.123889 14222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:20:16.125085 14222 hybrid_clock.cc:648] HybridClock initialized: now 1790572816125041 us; error 34 us; skew 500 ppm
I20260928 05:20:16.125384 14222 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:20:16 dist-test-slave-whsn krb5kdc[14209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572816, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:20:16.131088 14222 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 05:20:16.132375 14222 webserver.cc:460] Webserver started at http://127.0.97.62:40133/ using document root <none> and password file <none>
I20260928 05:20:16.133021 14222 fs_manager.cc:359] Metadata directory not provided
I20260928 05:20:16.133144 14222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:20:16.133407 14222 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:20:16.136144 14222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/master-0/data/instance:
uuid: "3b36d06442754a95908b39ef6b46ad06"
format_stamp: "Formatted at 2026-09-28 05:20:16 on dist-test-slave-whsn"
server_key: "e98dabea40fe88d3c480d40e6b9ff037"
server_key_iv: "77733407db60baee2278fc1927cf2899"
server_key_version: "encryptionkey@0"
I20260928 05:20:16.136832 14222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/master-0/wal/instance:
uuid: "3b36d06442754a95908b39ef6b46ad06"
format_stamp: "Formatted at 2026-09-28 05:20:16 on dist-test-slave-whsn"
server_key: "e98dabea40fe88d3c480d40e6b9ff037"
server_key_iv: "77733407db60baee2278fc1927cf2899"
server_key_version: "encryptionkey@0"
I20260928 05:20:16.141441 14222 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 05:20:16.144529 14237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:20:16.145814 14222 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:20:16.145949 14222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/master-0/wal
uuid: "3b36d06442754a95908b39ef6b46ad06"
format_stamp: "Formatted at 2026-09-28 05:20:16 on dist-test-slave-whsn"
server_key: "e98dabea40fe88d3c480d40e6b9ff037"
server_key_iv: "77733407db60baee2278fc1927cf2899"
server_key_version: "encryptionkey@0"
I20260928 05:20:16.146101 14222 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:20:16.175661 14222 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:20:16.179172 14222 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:20:16.179380 14222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:20:16.194554 14222 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38873
I20260928 05:20:16.194577 14288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38873 every 8 connection(s)
I20260928 05:20:16.195734 14222 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:20:16.199163 14289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:20:16.205852 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 14222
I20260928 05:20:16.206063 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/master-0/wal/instance
I20260928 05:20:16.206655 388 external_mini_cluster.cc:1428] Setting key c3a781c06ad4a2f9eeaafe2441b5da1d
I20260928 05:20:16.208772 14289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b36d06442754a95908b39ef6b46ad06: Bootstrap starting.
I20260928 05:20:16.212589 14289 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b36d06442754a95908b39ef6b46ad06: Neither blocks nor log segments found. Creating new log.
I20260928 05:20:16.213589 14289 log.cc:824] T 00000000000000000000000000000000 P 3b36d06442754a95908b39ef6b46ad06: Log is configured to *not* fsync() on all Append() calls
I20260928 05:20:16.216312 14289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b36d06442754a95908b39ef6b46ad06: No bootstrap required, opened a new log
Sep 28 05:20:16 dist-test-slave-whsn krb5kdc[14209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572815, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:20:16.224736 14289 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b36d06442754a95908b39ef6b46ad06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b36d06442754a95908b39ef6b46ad06" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38873 } }
I20260928 05:20:16.225178 14289 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b36d06442754a95908b39ef6b46ad06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:20:16.225286 14289 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b36d06442754a95908b39ef6b46ad06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b36d06442754a95908b39ef6b46ad06, State: Initialized, Role: FOLLOWER
I20260928 05:20:16.225853 14289 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b36d06442754a95908b39ef6b46ad06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b36d06442754a95908b39ef6b46ad06" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38873 } }
I20260928 05:20:16.226133 14289 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b36d06442754a95908b39ef6b46ad06 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:20:16.226284 14289 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b36d06442754a95908b39ef6b46ad06 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:20:16.226434 14289 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b36d06442754a95908b39ef6b46ad06 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:20:16.229498 14292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:16.208393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40934 (local address 127.0.97.62:38873)
0928 05:20:16.209165 (+ 772us) server_negotiation.cc:206] Beginning negotiation
0928 05:20:16.209181 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 05:20:16.209227 (+ 46us) server_negotiation.cc:407] Connection header received
0928 05:20:16.211458 (+ 2231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:20:16.211485 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:20:16.211944 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:20:16.212327 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:20:16.213631 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:16.215293 (+ 1662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:16.216564 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:16.217567 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:16.217739 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:16.221316 (+ 3577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:20:16.221326 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:20:16.221347 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:20:16.221392 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:20:16.225121 (+ 3729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:16.225874 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:16.225881 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:16.225889 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:16.226077 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:16.226475 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:16.226480 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:16.226482 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:16.227148 (+ 666us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:20:16.227429 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0928 05:20:16.227830 (+ 401us) server_negotiation.cc:299] Negotiation successful
0928 05:20:16.228235 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":306,"threads_started":1}
I20260928 05:20:16.229533 14289 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b36d06442754a95908b39ef6b46ad06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b36d06442754a95908b39ef6b46ad06" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38873 } }
I20260928 05:20:16.230008 14289 leader_election.cc:302] T 00000000000000000000000000000000 P 3b36d06442754a95908b39ef6b46ad06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b36d06442754a95908b39ef6b46ad06; no voters:
I20260928 05:20:16.230571 14289 leader_election.cc:288] T 00000000000000000000000000000000 P 3b36d06442754a95908b39ef6b46ad06 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:20:16.230705 14294 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b36d06442754a95908b39ef6b46ad06 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:20:16.231874 14294 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b36d06442754a95908b39ef6b46ad06 [term 1 LEADER]: Becoming Leader. State: Replica: 3b36d06442754a95908b39ef6b46ad06, State: Running, Role: LEADER
I20260928 05:20:16.232398 14294 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b36d06442754a95908b39ef6b46ad06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b36d06442754a95908b39ef6b46ad06" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38873 } }
I20260928 05:20:16.232631 14289 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b36d06442754a95908b39ef6b46ad06 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:20:16.234952 14295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b36d06442754a95908b39ef6b46ad06 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3b36d06442754a95908b39ef6b46ad06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b36d06442754a95908b39ef6b46ad06" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38873 } } }
I20260928 05:20:16.237669 14295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b36d06442754a95908b39ef6b46ad06 [sys.catalog]: This master's current role is: LEADER
I20260928 05:20:16.234933 14296 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b36d06442754a95908b39ef6b46ad06 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b36d06442754a95908b39ef6b46ad06. Latest consensus state: current_term: 1 leader_uuid: "3b36d06442754a95908b39ef6b46ad06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b36d06442754a95908b39ef6b46ad06" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38873 } } }
I20260928 05:20:16.238116 14296 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b36d06442754a95908b39ef6b46ad06 [sys.catalog]: This master's current role is: LEADER
I20260928 05:20:16.245740 14301 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:20:16.250044 14301 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:20:16.257730 14301 catalog_manager.cc:1347] Generated new cluster ID: 300d65fd8cab49eda278356e8e700e03
I20260928 05:20:16.257829 14301 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:20:16.271848 14301 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:20:16.273634 14301 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:20:16.290519 14301 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3b36d06442754a95908b39ef6b46ad06: Generated new TSK 0
I20260928 05:20:16.291481 14301 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260928 05:20:16.341339 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38873
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:20:16.645982 14315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:20:16.646256 14315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:20:16.646382 14315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:20:16.653525 14315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:20:16.653630 14315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:20:16.653766 14315 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 05:20:16.663012 14315 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38873
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:20:16.664254 14315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:20:16.665530 14315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:20:16.677631 14321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:20:16.677700 14323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:20:16.677676 14320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:20:16.678508 14315 server_base.cc:1034] running on GCE node
I20260928 05:20:16.679082 14315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:20:16.679919 14315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:20:16.681118 14315 hybrid_clock.cc:648] HybridClock initialized: now 1790572816681060 us; error 51 us; skew 500 ppm
I20260928 05:20:16.681398 14315 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:20:16 dist-test-slave-whsn krb5kdc[14209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572816, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:20:16.684286 14315 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260928 05:20:16.685576 14315 webserver.cc:460] Webserver started at http://127.0.97.1:42701/ using document root <none> and password file <none>
I20260928 05:20:16.686200 14315 fs_manager.cc:359] Metadata directory not provided
I20260928 05:20:16.686295 14315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:20:16.686522 14315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:20:16.689256 14315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-0/data/instance:
uuid: "d9fc7afd66a64bf997acf128e9f4ad28"
format_stamp: "Formatted at 2026-09-28 05:20:16 on dist-test-slave-whsn"
server_key: "abfff00c90c1ec802b746e533217ed74"
server_key_iv: "6bdbfa0c04b72d97b3fe600767c65655"
server_key_version: "encryptionkey@0"
I20260928 05:20:16.689949 14315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-0/wal/instance:
uuid: "d9fc7afd66a64bf997acf128e9f4ad28"
format_stamp: "Formatted at 2026-09-28 05:20:16 on dist-test-slave-whsn"
server_key: "abfff00c90c1ec802b746e533217ed74"
server_key_iv: "6bdbfa0c04b72d97b3fe600767c65655"
server_key_version: "encryptionkey@0"
I20260928 05:20:16.694535 14315 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 05:20:16.697623 14330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:20:16.698993 14315 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:20:16.699159 14315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-0/wal
uuid: "d9fc7afd66a64bf997acf128e9f4ad28"
format_stamp: "Formatted at 2026-09-28 05:20:16 on dist-test-slave-whsn"
server_key: "abfff00c90c1ec802b746e533217ed74"
server_key_iv: "6bdbfa0c04b72d97b3fe600767c65655"
server_key_version: "encryptionkey@0"
I20260928 05:20:16.699358 14315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:20:16.737615 14315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:20:16.741524 14315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:20:16.741776 14315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:20:16.742717 14315 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:20:16.743988 14315 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:20:16.744055 14315 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:20:16.744179 14315 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:20:16.744232 14315 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:20:16.773619 14315 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41813
I20260928 05:20:16.773660 14442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41813 every 8 connection(s)
I20260928 05:20:16.774807 14315 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-0/data/info.pb
I20260928 05:20:16.780206 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 14315
I20260928 05:20:16.780423 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-0/wal/instance
I20260928 05:20:16.781211 388 external_mini_cluster.cc:1428] Setting key 81d5da26baebc6aa015e4479183dc75e
Sep 28 05:20:16 dist-test-slave-whsn krb5kdc[14209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572816, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:20:16.794287 14445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:16.777197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50069 (local address 127.0.97.62:38873)
0928 05:20:16.777617 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0928 05:20:16.777622 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:20:16.778421 (+ 799us) server_negotiation.cc:407] Connection header received
0928 05:20:16.780270 (+ 1849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:20:16.780274 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:20:16.780354 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:20:16.780455 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:20:16.782307 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:16.783073 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:16.784530 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:16.785111 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:16.785219 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:16.788548 (+ 3329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:20:16.788553 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:20:16.788556 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:20:16.788587 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:20:16.791294 (+ 2707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:16.792027 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:16.792031 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:16.792034 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:16.792166 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:16.792537 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:16.792542 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:16.792545 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:16.792957 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:20:16.793087 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0928 05:20:16.793739 (+ 652us) server_negotiation.cc:299] Negotiation successful
0928 05:20:16.793955 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":221,"threads_started":1}
I20260928 05:20:16.795086 14446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:16.777637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38873 (local address 127.0.97.1:50069)
0928 05:20:16.778271 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:16.778312 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 05:20:16.779971 (+ 1659us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:20:16.780740 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:20:16.780751 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:20:16.781467 (+ 716us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:20:16.782142 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:20:16.782160 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:20:16.783203 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:20:16.783207 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:20:16.784391 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:20:16.784402 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:20:16.785288 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:20:16.785292 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:20:16.785481 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:16.786079 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:20:16.786099 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:20:16.788317 (+ 2218us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:20:16.791459 (+ 3142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:20:16.791466 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:20:16.791478 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:20:16.791916 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:20:16.792258 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:20:16.792261 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:20:16.792263 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:20:16.792444 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:20:16.793106 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:20:16.793112 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:20:16.793401 (+ 289us) client_negotiation.cc:769] Sending connection context
0928 05:20:16.793642 (+ 241us) client_negotiation.cc:240] Negotiation successful
0928 05:20:16.794008 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":251,"threads_started":1}
I20260928 05:20:16.796412 14443 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38873
I20260928 05:20:16.796700 14443 heartbeater.cc:464] Registering TS with master...
I20260928 05:20:16.797384 14443 heartbeater.cc:511] Master 127.0.97.62:38873 requested a full tablet report, sending...
I20260928 05:20:16.799278 14254 ts_manager.cc:194] Registered new tserver with Master: d9fc7afd66a64bf997acf128e9f4ad28 (127.0.97.1:41813)
I20260928 05:20:16.801273 14254 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:50069
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260928 05:20:16.827729 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38873
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:20:17.137038 14449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:20:17.137295 14449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:20:17.137405 14449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:20:17.144497 14449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:20:17.144603 14449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:20:17.144726 14449 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 05:20:17.154045 14449 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38873
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:20:17.155148 14449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:20:17.156351 14449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:20:17.168555 14455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:20:17.168640 14454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:20:17.168601 14457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:20:17.169260 14449 server_base.cc:1034] running on GCE node
I20260928 05:20:17.169783 14449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:20:17.170557 14449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:20:17.171741 14449 hybrid_clock.cc:648] HybridClock initialized: now 1790572817171689 us; error 45 us; skew 500 ppm
I20260928 05:20:17.172017 14449 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:20:17 dist-test-slave-whsn krb5kdc[14209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572817, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:20:17.174881 14449 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260928 05:20:17.176134 14449 webserver.cc:460] Webserver started at http://127.0.97.2:37261/ using document root <none> and password file <none>
I20260928 05:20:17.176697 14449 fs_manager.cc:359] Metadata directory not provided
I20260928 05:20:17.176789 14449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:20:17.177069 14449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:20:17.179610 14449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-1/data/instance:
uuid: "3c747673187c42a894b270c6769ddab2"
format_stamp: "Formatted at 2026-09-28 05:20:17 on dist-test-slave-whsn"
server_key: "b3f72ace898d07ce94f3ff18a606b377"
server_key_iv: "dc2d72eee7b3063c3403165fe5fbdcf1"
server_key_version: "encryptionkey@0"
I20260928 05:20:17.180270 14449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-1/wal/instance:
uuid: "3c747673187c42a894b270c6769ddab2"
format_stamp: "Formatted at 2026-09-28 05:20:17 on dist-test-slave-whsn"
server_key: "b3f72ace898d07ce94f3ff18a606b377"
server_key_iv: "dc2d72eee7b3063c3403165fe5fbdcf1"
server_key_version: "encryptionkey@0"
I20260928 05:20:17.184669 14449 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260928 05:20:17.187721 14464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:20:17.188889 14449 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 05:20:17.189033 14449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-1/wal
uuid: "3c747673187c42a894b270c6769ddab2"
format_stamp: "Formatted at 2026-09-28 05:20:17 on dist-test-slave-whsn"
server_key: "b3f72ace898d07ce94f3ff18a606b377"
server_key_iv: "dc2d72eee7b3063c3403165fe5fbdcf1"
server_key_version: "encryptionkey@0"
I20260928 05:20:17.189184 14449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:20:17.210902 14449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:20:17.214738 14449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:20:17.214982 14449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:20:17.215801 14449 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:20:17.217000 14449 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:20:17.217068 14449 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:20:17.217177 14449 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:20:17.217224 14449 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:20:17.245759 14449 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37923
I20260928 05:20:17.245811 14576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37923 every 8 connection(s)
I20260928 05:20:17.247023 14449 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-1/data/info.pb
I20260928 05:20:17.255411 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 14449
I20260928 05:20:17.255596 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-1/wal/instance
I20260928 05:20:17.256107 388 external_mini_cluster.cc:1428] Setting key 99dd00e4a3a72de4bed9d5328c2c995d
Sep 28 05:20:17 dist-test-slave-whsn krb5kdc[14209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572817, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:20:17.265161 14445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:17.249247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33971 (local address 127.0.97.62:38873)
0928 05:20:17.249487 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0928 05:20:17.249493 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:20:17.250584 (+ 1091us) server_negotiation.cc:407] Connection header received
0928 05:20:17.252280 (+ 1696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:20:17.252284 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:20:17.252348 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:20:17.252438 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:20:17.253802 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:17.254543 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:17.256432 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:17.256837 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:17.256971 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:17.260061 (+ 3090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:20:17.260067 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:20:17.260071 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:20:17.260099 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:20:17.262527 (+ 2428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:17.263152 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:17.263157 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:17.263161 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:17.263282 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:17.263647 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:17.263652 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:17.263655 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:17.263893 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:20:17.264017 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 05:20:17.264611 (+ 594us) server_negotiation.cc:299] Negotiation successful
0928 05:20:17.264828 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 05:20:17.266014 14579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:17.249739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38873 (local address 127.0.97.2:33971)
0928 05:20:17.250407 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:17.250495 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0928 05:20:17.252040 (+ 1545us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:20:17.252603 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:20:17.252610 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:20:17.253115 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:20:17.253633 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:20:17.253650 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:20:17.254674 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:20:17.254677 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:20:17.256308 (+ 1631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:20:17.256322 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:20:17.257063 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:20:17.257071 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:20:17.257269 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:17.257899 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:20:17.257920 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:20:17.259870 (+ 1950us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:20:17.262647 (+ 2777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:20:17.262655 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:20:17.262669 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:20:17.263050 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:20:17.263376 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:20:17.263380 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:20:17.263384 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:20:17.263548 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:20:17.264008 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:20:17.264015 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:20:17.264290 (+ 275us) client_negotiation.cc:769] Sending connection context
0928 05:20:17.264507 (+ 217us) client_negotiation.cc:240] Negotiation successful
0928 05:20:17.264991 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":258,"threads_started":1}
I20260928 05:20:17.267251 14577 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38873
I20260928 05:20:17.267547 14577 heartbeater.cc:464] Registering TS with master...
I20260928 05:20:17.268307 14577 heartbeater.cc:511] Master 127.0.97.62:38873 requested a full tablet report, sending...
I20260928 05:20:17.269707 14254 ts_manager.cc:194] Registered new tserver with Master: 3c747673187c42a894b270c6769ddab2 (127.0.97.2:37923)
I20260928 05:20:17.270491 14254 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:33971
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260928 05:20:17.299347 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38873
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:20:17.593925 14582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:20:17.594147 14582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:20:17.594267 14582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:20:17.601032 14582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:20:17.601123 14582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:20:17.601267 14582 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 05:20:17.610582 14582 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38873
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:20:17.611730 14582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:20:17.612982 14582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:20:17.624817 14587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:20:17.624878 14588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:20:17.624915 14590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:20:17.625677 14582 server_base.cc:1034] running on GCE node
I20260928 05:20:17.626209 14582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:20:17.627043 14582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:20:17.628237 14582 hybrid_clock.cc:648] HybridClock initialized: now 1790572817628192 us; error 39 us; skew 500 ppm
I20260928 05:20:17.628507 14582 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:20:17 dist-test-slave-whsn krb5kdc[14209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572817, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:20:17.631440 14582 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260928 05:20:17.632706 14582 webserver.cc:460] Webserver started at http://127.0.97.3:41963/ using document root <none> and password file <none>
I20260928 05:20:17.633327 14582 fs_manager.cc:359] Metadata directory not provided
I20260928 05:20:17.633421 14582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:20:17.633662 14582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:20:17.636401 14582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-2/data/instance:
uuid: "48927739d82c4f6a8274019312c386c6"
format_stamp: "Formatted at 2026-09-28 05:20:17 on dist-test-slave-whsn"
server_key: "a9c2679720605cdf6dedaff6d97079d3"
server_key_iv: "87e381d6de55e57ee2203c2d2cd41123"
server_key_version: "encryptionkey@0"
I20260928 05:20:17.637141 14582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-2/wal/instance:
uuid: "48927739d82c4f6a8274019312c386c6"
format_stamp: "Formatted at 2026-09-28 05:20:17 on dist-test-slave-whsn"
server_key: "a9c2679720605cdf6dedaff6d97079d3"
server_key_iv: "87e381d6de55e57ee2203c2d2cd41123"
server_key_version: "encryptionkey@0"
I20260928 05:20:17.641916 14582 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 05:20:17.645241 14597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:20:17.646510 14582 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260928 05:20:17.646646 14582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-2/wal
uuid: "48927739d82c4f6a8274019312c386c6"
format_stamp: "Formatted at 2026-09-28 05:20:17 on dist-test-slave-whsn"
server_key: "a9c2679720605cdf6dedaff6d97079d3"
server_key_iv: "87e381d6de55e57ee2203c2d2cd41123"
server_key_version: "encryptionkey@0"
I20260928 05:20:17.646803 14582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:20:17.674861 14582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:20:17.678725 14582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:20:17.678941 14582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:20:17.679764 14582 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:20:17.681001 14582 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:20:17.681070 14582 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:20:17.681154 14582 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:20:17.681201 14582 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:20:17.708712 14582 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33283
I20260928 05:20:17.708752 14709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33283 every 8 connection(s)
I20260928 05:20:17.709894 14582 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-2/data/info.pb
I20260928 05:20:17.715924 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 14582
I20260928 05:20:17.716109 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-2/wal/instance
I20260928 05:20:17.716645 388 external_mini_cluster.cc:1428] Setting key 83e84dbd0a4a76f547c785dcf35a53f9
Sep 28 05:20:17 dist-test-slave-whsn krb5kdc[14209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572817, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:20:17.727730 14445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:17.712186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45193 (local address 127.0.97.62:38873)
0928 05:20:17.712383 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 05:20:17.712390 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:20:17.713484 (+ 1094us) server_negotiation.cc:407] Connection header received
0928 05:20:17.715115 (+ 1631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:20:17.715120 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:20:17.715183 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:20:17.715265 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:20:17.717019 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:17.718000 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:17.719788 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:17.720184 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:17.720271 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:17.723241 (+ 2970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:20:17.723246 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:20:17.723250 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:20:17.723281 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:20:17.725446 (+ 2165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:17.726033 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:17.726036 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:17.726039 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:17.726154 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:17.726475 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:17.726478 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:17.726480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:17.726726 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:20:17.726827 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 05:20:17.727350 (+ 523us) server_negotiation.cc:299] Negotiation successful
0928 05:20:17.727500 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 05:20:17.728551 14712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:17.712667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38873 (local address 127.0.97.3:45193)
0928 05:20:17.713327 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:17.713371 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 05:20:17.714894 (+ 1523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:20:17.715441 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:20:17.715448 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:20:17.716080 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:20:17.716826 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:20:17.716870 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:20:17.718147 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:20:17.718152 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:20:17.719666 (+ 1514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:20:17.719680 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:20:17.720343 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:20:17.720347 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:20:17.720539 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:17.721101 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:20:17.721121 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:20:17.723064 (+ 1943us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:20:17.725568 (+ 2504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:20:17.725574 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:20:17.725584 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:20:17.725943 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:20:17.726238 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:20:17.726241 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:20:17.726243 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:20:17.726399 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:20:17.726808 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:20:17.726813 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:20:17.727074 (+ 261us) client_negotiation.cc:769] Sending connection context
0928 05:20:17.727271 (+ 197us) client_negotiation.cc:240] Negotiation successful
0928 05:20:17.727610 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":283,"threads_started":1}
I20260928 05:20:17.729725 14710 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38873
I20260928 05:20:17.729967 14710 heartbeater.cc:464] Registering TS with master...
I20260928 05:20:17.730692 14710 heartbeater.cc:511] Master 127.0.97.62:38873 requested a full tablet report, sending...
I20260928 05:20:17.732018 14254 ts_manager.cc:194] Registered new tserver with Master: 48927739d82c4f6a8274019312c386c6 (127.0.97.3:33283)
I20260928 05:20:17.732766 14254 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:45193
I20260928 05:20:17.743050 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 05:20:17.758649 14445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:17.747121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40942 (local address 127.0.97.62:38873)
0928 05:20:17.747265 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 05:20:17.747271 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:20:17.747670 (+ 399us) server_negotiation.cc:407] Connection header received
0928 05:20:17.747816 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:20:17.747821 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:20:17.747887 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:20:17.747962 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:20:17.749362 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:17.750097 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:17.751448 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:17.751839 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:17.751938 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:17.754108 (+ 2170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:20:17.754112 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:20:17.754115 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:20:17.754143 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:20:17.756248 (+ 2105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:17.756838 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:17.756843 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:17.756847 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:17.756983 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:17.757316 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:17.757321 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:17.757325 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:17.757613 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:20:17.757827 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0928 05:20:17.758185 (+ 358us) server_negotiation.cc:299] Negotiation successful
0928 05:20:17.758385 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 05:20:17.761885 14254 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40942:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 05:20:17.768356 14254 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 05:20:17.801523 14727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:17.781486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54558 (local address 127.0.97.2:37923)
0928 05:20:17.789541 (+ 8055us) server_negotiation.cc:206] Beginning negotiation
0928 05:20:17.789548 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:20:17.793878 (+ 4330us) server_negotiation.cc:407] Connection header received
0928 05:20:17.794032 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:20:17.794039 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:20:17.794255 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:20:17.794425 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:20:17.795131 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:17.796346 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:17.798963 (+ 2617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:17.800226 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:17.800771 (+ 545us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:17.800998 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0928 05:20:17.801066 (+ 68us) server_negotiation.cc:299] Negotiation successful
0928 05:20:17.801223 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7917,"thread_start_us":7802,"threads_started":1}
I20260928 05:20:17.801738 14724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:17.779960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33283 (local address 127.0.0.1:41358)
0928 05:20:17.791708 (+ 11748us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:17.791761 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 05:20:17.791934 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:20:17.794921 (+ 2987us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:20:17.794925 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:20:17.794948 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:20:17.795455 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:20:17.795467 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:20:17.797268 (+ 1801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:20:17.797275 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:20:17.798913 (+ 1638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:20:17.798929 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:20:17.800525 (+ 1596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:20:17.800531 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:20:17.800751 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:17.800818 (+ 67us) client_negotiation.cc:769] Sending connection context
0928 05:20:17.800954 (+ 136us) client_negotiation.cc:240] Negotiation successful
0928 05:20:17.801196 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11496,"mutex_wait_us":115,"thread_start_us":13053,"threads_started":1}
I20260928 05:20:17.801780 14725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:17.780595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37923 (local address 127.0.0.1:54558)
0928 05:20:17.793779 (+ 13184us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:17.793799 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 05:20:17.793901 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:20:17.794456 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:20:17.794462 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:20:17.794491 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:20:17.794914 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:20:17.794925 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:20:17.796483 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:20:17.796493 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:20:17.798784 (+ 2291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:20:17.798802 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:20:17.800458 (+ 1656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:20:17.800464 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:20:17.800679 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:17.800822 (+ 143us) client_negotiation.cc:769] Sending connection context
0928 05:20:17.800962 (+ 140us) client_negotiation.cc:240] Negotiation successful
0928 05:20:17.801196 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13062,"mutex_wait_us":87,"thread_start_us":12927,"threads_started":1}
I20260928 05:20:17.802310 14726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:17.781181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49742 (local address 127.0.97.1:41813)
0928 05:20:17.789147 (+ 7966us) server_negotiation.cc:206] Beginning negotiation
0928 05:20:17.789156 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:20:17.793389 (+ 4233us) server_negotiation.cc:407] Connection header received
0928 05:20:17.794013 (+ 624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:20:17.794023 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:20:17.794283 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:20:17.794477 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:20:17.795412 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:17.796970 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:17.799487 (+ 2517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:17.800840 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:17.800978 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:17.801185 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0928 05:20:17.801835 (+ 650us) server_negotiation.cc:299] Negotiation successful
0928 05:20:17.801984 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7796,"thread_start_us":7667,"threads_started":1}
I20260928 05:20:17.802637 14722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:17.780370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41813 (local address 127.0.0.1:49742)
0928 05:20:17.793265 (+ 12895us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:17.793286 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 05:20:17.793791 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:20:17.794703 (+ 912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:20:17.794708 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:20:17.794736 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:20:17.795226 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:20:17.795238 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:20:17.797135 (+ 1897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:20:17.797140 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:20:17.799320 (+ 2180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:20:17.799333 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:20:17.801321 (+ 1988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:20:17.801327 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:20:17.801484 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:17.801514 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 05:20:17.801607 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 05:20:17.801739 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12782,"mutex_wait_us":58,"thread_start_us":10952,"threads_started":1}
I20260928 05:20:17.803040 14723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:17.780289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41358 (local address 127.0.97.3:33283)
0928 05:20:17.794401 (+ 14112us) server_negotiation.cc:206] Beginning negotiation
0928 05:20:17.794409 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:20:17.794439 (+ 30us) server_negotiation.cc:407] Connection header received
0928 05:20:17.794519 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:20:17.794526 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:20:17.794734 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:20:17.794912 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:20:17.795623 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:17.797116 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:17.799073 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:17.800282 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:17.802251 (+ 1969us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:17.802457 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0928 05:20:17.802520 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 05:20:17.802735 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13941,"thread_start_us":14299,"threads_started":1}
I20260928 05:20:17.805630 14512 tablet_service.cc:1467] Processing CreateTablet for tablet f4d5b7b89ba34cb09e1e5dc27d26f00a (DEFAULT_TABLE table=test-table [id=5c238e004a4a4b999172582cd9a430b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:20:17.806406 14443 heartbeater.cc:503] Master 127.0.97.62:38873 was elected leader, sending a full tablet report...
I20260928 05:20:17.806208 14378 tablet_service.cc:1467] Processing CreateTablet for tablet f4d5b7b89ba34cb09e1e5dc27d26f00a (DEFAULT_TABLE table=test-table [id=5c238e004a4a4b999172582cd9a430b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:20:17.806941 14512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4d5b7b89ba34cb09e1e5dc27d26f00a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:20:17.807540 14378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4d5b7b89ba34cb09e1e5dc27d26f00a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:20:17.807281 14645 tablet_service.cc:1467] Processing CreateTablet for tablet f4d5b7b89ba34cb09e1e5dc27d26f00a (DEFAULT_TABLE table=test-table [id=5c238e004a4a4b999172582cd9a430b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:20:17.808729 14645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4d5b7b89ba34cb09e1e5dc27d26f00a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:20:17.817538 14728 tablet_bootstrap.cc:492] T f4d5b7b89ba34cb09e1e5dc27d26f00a P 3c747673187c42a894b270c6769ddab2: Bootstrap starting.
I20260928 05:20:17.820621 14728 tablet_bootstrap.cc:654] T f4d5b7b89ba34cb09e1e5dc27d26f00a P 3c747673187c42a894b270c6769ddab2: Neither blocks nor log segments found. Creating new log.
I20260928 05:20:17.820837 14729 tablet_bootstrap.cc:492] T f4d5b7b89ba34cb09e1e5dc27d26f00a P d9fc7afd66a64bf997acf128e9f4ad28: Bootstrap starting.
I20260928 05:20:17.822105 14730 tablet_bootstrap.cc:492] T f4d5b7b89ba34cb09e1e5dc27d26f00a P 48927739d82c4f6a8274019312c386c6: Bootstrap starting.
I20260928 05:20:17.822865 14728 log.cc:824] T f4d5b7b89ba34cb09e1e5dc27d26f00a P 3c747673187c42a894b270c6769ddab2: Log is configured to *not* fsync() on all Append() calls
I20260928 05:20:17.823951 14729 tablet_bootstrap.cc:654] T f4d5b7b89ba34cb09e1e5dc27d26f00a P d9fc7afd66a64bf997acf128e9f4ad28: Neither blocks nor log segments found. Creating new log.
I20260928 05:20:17.824841 14730 tablet_bootstrap.cc:654] T f4d5b7b89ba34cb09e1e5dc27d26f00a P 48927739d82c4f6a8274019312c386c6: Neither blocks nor log segments found. Creating new log.
I20260928 05:20:17.825085 14729 log.cc:824] T f4d5b7b89ba34cb09e1e5dc27d26f00a P d9fc7afd66a64bf997acf128e9f4ad28: Log is configured to *not* fsync() on all Append() calls
I20260928 05:20:17.825803 14728 tablet_bootstrap.cc:492] T f4d5b7b89ba34cb09e1e5dc27d26f00a P 3c747673187c42a894b270c6769ddab2: No bootstrap required, opened a new log
I20260928 05:20:17.825872 14730 log.cc:824] T f4d5b7b89ba34cb09e1e5dc27d26f00a P 48927739d82c4f6a8274019312c386c6: Log is configured to *not* fsync() on all Append() calls
I20260928 05:20:17.826042 14728 ts_tablet_manager.cc:1397] T f4d5b7b89ba34cb09e1e5dc27d26f00a P 3c747673187c42a894b270c6769ddab2: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260928 05:20:17.827865 14729 tablet_bootstrap.cc:492] T f4d5b7b89ba34cb09e1e5dc27d26f00a P d9fc7afd66a64bf997acf128e9f4ad28: No bootstrap required, opened a new log
I20260928 05:20:17.828162 14729 ts_tablet_manager.cc:1397] T f4d5b7b89ba34cb09e1e5dc27d26f00a P d9fc7afd66a64bf997acf128e9f4ad28: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.007s
I20260928 05:20:17.830856 14730 tablet_bootstrap.cc:492] T f4d5b7b89ba34cb09e1e5dc27d26f00a P 48927739d82c4f6a8274019312c386c6: No bootstrap required, opened a new log
I20260928 05:20:17.831069 14730 ts_tablet_manager.cc:1397] T f4d5b7b89ba34cb09e1e5dc27d26f00a P 48927739d82c4f6a8274019312c386c6: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20260928 05:20:17.836305 14728 raft_consensus.cc:348] T f4d5b7b89ba34cb09e1e5dc27d26f00a P 3c747673187c42a894b270c6769ddab2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48927739d82c4f6a8274019312c386c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33283 } } peers { permanent_uuid: "d9fc7afd66a64bf997acf128e9f4ad28" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41813 } } peers { permanent_uuid: "3c747673187c42a894b270c6769ddab2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37923 } }
I20260928 05:20:17.836724 14728 raft_consensus.cc:374] T f4d5b7b89ba34cb09e1e5dc27d26f00a P 3c747673187c42a894b270c6769ddab2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:20:17.836813 14728 raft_consensus.cc:729] T f4d5b7b89ba34cb09e1e5dc27d26f00a P 3c747673187c42a894b270c6769ddab2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c747673187c42a894b270c6769ddab2, State: Initialized, Role: FOLLOWER
I20260928 05:20:17.837394 14728 consensus_queue.cc:260] T f4d5b7b89ba34cb09e1e5dc27d26f00a P 3c747673187c42a894b270c6769ddab2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48927739d82c4f6a8274019312c386c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33283 } } peers { permanent_uuid: "d9fc7afd66a64bf997acf128e9f4ad28" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41813 } } peers { permanent_uuid: "3c747673187c42a894b270c6769ddab2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37923 } }
I20260928 05:20:17.837599 14730 raft_consensus.cc:348] T f4d5b7b89ba34cb09e1e5dc27d26f00a P 48927739d82c4f6a8274019312c386c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48927739d82c4f6a8274019312c386c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33283 } } peers { permanent_uuid: "d9fc7afd66a64bf997acf128e9f4ad28" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41813 } } peers { permanent_uuid: "3c747673187c42a894b270c6769ddab2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37923 } }
I20260928 05:20:17.838001 14730 raft_consensus.cc:374] T f4d5b7b89ba34cb09e1e5dc27d26f00a P 48927739d82c4f6a8274019312c386c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:20:17.837862 14729 raft_consensus.cc:348] T f4d5b7b89ba34cb09e1e5dc27d26f00a P d9fc7afd66a64bf997acf128e9f4ad28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48927739d82c4f6a8274019312c386c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33283 } } peers { permanent_uuid: "d9fc7afd66a64bf997acf128e9f4ad28" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41813 } } peers { permanent_uuid: "3c747673187c42a894b270c6769ddab2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37923 } }
I20260928 05:20:17.838131 14730 raft_consensus.cc:729] T f4d5b7b89ba34cb09e1e5dc27d26f00a P 48927739d82c4f6a8274019312c386c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48927739d82c4f6a8274019312c386c6, State: Initialized, Role: FOLLOWER
I20260928 05:20:17.838312 14729 raft_consensus.cc:374] T f4d5b7b89ba34cb09e1e5dc27d26f00a P d9fc7afd66a64bf997acf128e9f4ad28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:20:17.838444 14577 heartbeater.cc:503] Master 127.0.97.62:38873 was elected leader, sending a full tablet report...
I20260928 05:20:17.838449 14729 raft_consensus.cc:729] T f4d5b7b89ba34cb09e1e5dc27d26f00a P d9fc7afd66a64bf997acf128e9f4ad28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9fc7afd66a64bf997acf128e9f4ad28, State: Initialized, Role: FOLLOWER
I20260928 05:20:17.838783 14730 consensus_queue.cc:260] T f4d5b7b89ba34cb09e1e5dc27d26f00a P 48927739d82c4f6a8274019312c386c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48927739d82c4f6a8274019312c386c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33283 } } peers { permanent_uuid: "d9fc7afd66a64bf997acf128e9f4ad28" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41813 } } peers { permanent_uuid: "3c747673187c42a894b270c6769ddab2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37923 } }
I20260928 05:20:17.839066 14728 ts_tablet_manager.cc:1428] T f4d5b7b89ba34cb09e1e5dc27d26f00a P 3c747673187c42a894b270c6769ddab2: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
I20260928 05:20:17.838945 14729 consensus_queue.cc:260] T f4d5b7b89ba34cb09e1e5dc27d26f00a P d9fc7afd66a64bf997acf128e9f4ad28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48927739d82c4f6a8274019312c386c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33283 } } peers { permanent_uuid: "d9fc7afd66a64bf997acf128e9f4ad28" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41813 } } peers { permanent_uuid: "3c747673187c42a894b270c6769ddab2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37923 } }
I20260928 05:20:17.839931 14710 heartbeater.cc:503] Master 127.0.97.62:38873 was elected leader, sending a full tablet report...
I20260928 05:20:17.840528 14730 ts_tablet_manager.cc:1428] T f4d5b7b89ba34cb09e1e5dc27d26f00a P 48927739d82c4f6a8274019312c386c6: Time spent starting tablet: real 0.009s user 0.008s sys 0.002s
I20260928 05:20:17.843770 14729 ts_tablet_manager.cc:1428] T f4d5b7b89ba34cb09e1e5dc27d26f00a P d9fc7afd66a64bf997acf128e9f4ad28: Time spent starting tablet: real 0.015s user 0.011s sys 0.003s
W20260928 05:20:17.961333 14711 tablet.cc:2369] T f4d5b7b89ba34cb09e1e5dc27d26f00a P 48927739d82c4f6a8274019312c386c6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 05:20:17.998792 14578 tablet.cc:2369] T f4d5b7b89ba34cb09e1e5dc27d26f00a P 3c747673187c42a894b270c6769ddab2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 05:20:18.026950 14444 tablet.cc:2369] T f4d5b7b89ba34cb09e1e5dc27d26f00a P d9fc7afd66a64bf997acf128e9f4ad28: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 05:20:18.084082 14734 raft_consensus.cc:482] T f4d5b7b89ba34cb09e1e5dc27d26f00a P 3c747673187c42a894b270c6769ddab2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:20:18.084349 14734 raft_consensus.cc:504] T f4d5b7b89ba34cb09e1e5dc27d26f00a P 3c747673187c42a894b270c6769ddab2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48927739d82c4f6a8274019312c386c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33283 } } peers { permanent_uuid: "d9fc7afd66a64bf997acf128e9f4ad28" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41813 } } peers { permanent_uuid: "3c747673187c42a894b270c6769ddab2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37923 } }
I20260928 05:20:18.085896 14734 leader_election.cc:288] T f4d5b7b89ba34cb09e1e5dc27d26f00a P 3c747673187c42a894b270c6769ddab2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 48927739d82c4f6a8274019312c386c6 (127.0.97.3:33283), d9fc7afd66a64bf997acf128e9f4ad28 (127.0.97.1:41813)
I20260928 05:20:18.094197 14726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:18.086464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36811 (local address 127.0.97.1:41813)
0928 05:20:18.086674 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 05:20:18.086680 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:20:18.087362 (+ 682us) server_negotiation.cc:407] Connection header received
0928 05:20:18.087487 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:20:18.087491 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:20:18.087560 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:20:18.087647 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:20:18.088260 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:18.089293 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:18.091944 (+ 2651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:18.093155 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:18.093282 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:18.093381 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 05:20:18.093835 (+ 454us) server_negotiation.cc:299] Negotiation successful
0928 05:20:18.093954 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 05:20:18.094198 14723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:18.086331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49271 (local address 127.0.97.3:33283)
0928 05:20:18.086522 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 05:20:18.086529 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:20:18.087152 (+ 623us) server_negotiation.cc:407] Connection header received
0928 05:20:18.087339 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:20:18.087344 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:20:18.087417 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:20:18.087497 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:20:18.088117 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:18.089236 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:18.091944 (+ 2708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:18.093165 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:18.093282 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:18.093381 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 05:20:18.093835 (+ 454us) server_negotiation.cc:299] Negotiation successful
0928 05:20:18.093954 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 05:20:18.094431 14737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:18.086323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41813 (local address 127.0.97.2:36811)
0928 05:20:18.087274 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:18.087292 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 05:20:18.087391 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:20:18.087672 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:20:18.087676 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:20:18.087699 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:20:18.088121 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:20:18.088132 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:20:18.089392 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:20:18.089396 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:20:18.091797 (+ 2401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:20:18.091812 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:20:18.093381 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:20:18.093388 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:20:18.093654 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:18.093720 (+ 66us) client_negotiation.cc:769] Sending connection context
0928 05:20:18.093812 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 05:20:18.093966 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":826,"mutex_wait_us":179,"thread_start_us":361,"threads_started":1}
I20260928 05:20:18.094452 14738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:18.086098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33283 (local address 127.0.97.2:49271)
0928 05:20:18.087053 (+ 955us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:18.087069 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 05:20:18.087207 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:20:18.087525 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:20:18.087529 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:20:18.087547 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:20:18.087963 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:20:18.087975 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:20:18.089350 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:20:18.089356 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:20:18.091806 (+ 2450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:20:18.091814 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:20:18.093393 (+ 1579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:20:18.093398 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:20:18.093653 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:18.093729 (+ 76us) client_negotiation.cc:769] Sending connection context
0928 05:20:18.093812 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 05:20:18.093966 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":798,"mutex_wait_us":189,"thread_start_us":869,"threads_started":1}
I20260928 05:20:18.095448 14398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4d5b7b89ba34cb09e1e5dc27d26f00a" candidate_uuid: "3c747673187c42a894b270c6769ddab2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9fc7afd66a64bf997acf128e9f4ad28" is_pre_election: true
I20260928 05:20:18.095482 14665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4d5b7b89ba34cb09e1e5dc27d26f00a" candidate_uuid: "3c747673187c42a894b270c6769ddab2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48927739d82c4f6a8274019312c386c6" is_pre_election: true
I20260928 05:20:18.095897 14665 raft_consensus.cc:2454] T f4d5b7b89ba34cb09e1e5dc27d26f00a P 48927739d82c4f6a8274019312c386c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c747673187c42a894b270c6769ddab2 in term 0.
I20260928 05:20:18.095897 14398 raft_consensus.cc:2454] T f4d5b7b89ba34cb09e1e5dc27d26f00a P d9fc7afd66a64bf997acf128e9f4ad28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c747673187c42a894b270c6769ddab2 in term 0.
I20260928 05:20:18.096468 14467 leader_election.cc:302] T f4d5b7b89ba34cb09e1e5dc27d26f00a P 3c747673187c42a894b270c6769ddab2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c747673187c42a894b270c6769ddab2, d9fc7afd66a64bf997acf128e9f4ad28; no voters:
I20260928 05:20:18.096802 14734 raft_consensus.cc:2781] T f4d5b7b89ba34cb09e1e5dc27d26f00a P 3c747673187c42a894b270c6769ddab2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 05:20:18.096963 14734 raft_consensus.cc:482] T f4d5b7b89ba34cb09e1e5dc27d26f00a P 3c747673187c42a894b270c6769ddab2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 05:20:18.097059 14734 raft_consensus.cc:3037] T f4d5b7b89ba34cb09e1e5dc27d26f00a P 3c747673187c42a894b270c6769ddab2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:20:18.099082 14734 raft_consensus.cc:504] T f4d5b7b89ba34cb09e1e5dc27d26f00a P 3c747673187c42a894b270c6769ddab2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48927739d82c4f6a8274019312c386c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33283 } } peers { permanent_uuid: "d9fc7afd66a64bf997acf128e9f4ad28" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41813 } } peers { permanent_uuid: "3c747673187c42a894b270c6769ddab2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37923 } }
I20260928 05:20:18.099586 14734 leader_election.cc:288] T f4d5b7b89ba34cb09e1e5dc27d26f00a P 3c747673187c42a894b270c6769ddab2 [CANDIDATE]: Term 1 election: Requested vote from peers 48927739d82c4f6a8274019312c386c6 (127.0.97.3:33283), d9fc7afd66a64bf997acf128e9f4ad28 (127.0.97.1:41813)
I20260928 05:20:18.099951 14665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4d5b7b89ba34cb09e1e5dc27d26f00a" candidate_uuid: "3c747673187c42a894b270c6769ddab2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48927739d82c4f6a8274019312c386c6"
I20260928 05:20:18.099980 14398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4d5b7b89ba34cb09e1e5dc27d26f00a" candidate_uuid: "3c747673187c42a894b270c6769ddab2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9fc7afd66a64bf997acf128e9f4ad28"
I20260928 05:20:18.100126 14665 raft_consensus.cc:3037] T f4d5b7b89ba34cb09e1e5dc27d26f00a P 48927739d82c4f6a8274019312c386c6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:20:18.100136 14398 raft_consensus.cc:3037] T f4d5b7b89ba34cb09e1e5dc27d26f00a P d9fc7afd66a64bf997acf128e9f4ad28 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:20:18.102203 14398 raft_consensus.cc:2454] T f4d5b7b89ba34cb09e1e5dc27d26f00a P d9fc7afd66a64bf997acf128e9f4ad28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c747673187c42a894b270c6769ddab2 in term 1.
I20260928 05:20:18.102268 14665 raft_consensus.cc:2454] T f4d5b7b89ba34cb09e1e5dc27d26f00a P 48927739d82c4f6a8274019312c386c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c747673187c42a894b270c6769ddab2 in term 1.
I20260928 05:20:18.102541 14467 leader_election.cc:302] T f4d5b7b89ba34cb09e1e5dc27d26f00a P 3c747673187c42a894b270c6769ddab2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c747673187c42a894b270c6769ddab2, d9fc7afd66a64bf997acf128e9f4ad28; no voters:
I20260928 05:20:18.102768 14734 raft_consensus.cc:2781] T f4d5b7b89ba34cb09e1e5dc27d26f00a P 3c747673187c42a894b270c6769ddab2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:20:18.103122 14734 raft_consensus.cc:686] T f4d5b7b89ba34cb09e1e5dc27d26f00a P 3c747673187c42a894b270c6769ddab2 [term 1 LEADER]: Becoming Leader. State: Replica: 3c747673187c42a894b270c6769ddab2, State: Running, Role: LEADER
I20260928 05:20:18.103574 14734 consensus_queue.cc:237] T f4d5b7b89ba34cb09e1e5dc27d26f00a P 3c747673187c42a894b270c6769ddab2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48927739d82c4f6a8274019312c386c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33283 } } peers { permanent_uuid: "d9fc7afd66a64bf997acf128e9f4ad28" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41813 } } peers { permanent_uuid: "3c747673187c42a894b270c6769ddab2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37923 } }
I20260928 05:20:18.108003 14253 catalog_manager.cc:5495] T f4d5b7b89ba34cb09e1e5dc27d26f00a P 3c747673187c42a894b270c6769ddab2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c747673187c42a894b270c6769ddab2 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "3c747673187c42a894b270c6769ddab2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48927739d82c4f6a8274019312c386c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d9fc7afd66a64bf997acf128e9f4ad28" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c747673187c42a894b270c6769ddab2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37923 } health_report { overall_health: HEALTHY } } }
I20260928 05:20:18.162832 14727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:18.158251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54562 (local address 127.0.97.2:37923)
0928 05:20:18.158462 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 05:20:18.158466 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:20:18.158482 (+ 16us) server_negotiation.cc:407] Connection header received
0928 05:20:18.158640 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:20:18.158644 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:20:18.158726 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:20:18.158837 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:20:18.159121 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:18.159793 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:18.161081 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:18.161451 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:18.161533 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:18.161903 (+ 370us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:20:18.162366 (+ 463us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:20:18.162464 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 05:20:18.162503 (+ 39us) server_negotiation.cc:299] Negotiation successful
0928 05:20:18.162561 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 05:20:18.170500 14665 raft_consensus.cc:1261] T f4d5b7b89ba34cb09e1e5dc27d26f00a P 48927739d82c4f6a8274019312c386c6 [term 1 FOLLOWER]: Refusing update from remote peer 3c747673187c42a894b270c6769ddab2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:20:18.170504 14398 raft_consensus.cc:1261] T f4d5b7b89ba34cb09e1e5dc27d26f00a P d9fc7afd66a64bf997acf128e9f4ad28 [term 1 FOLLOWER]: Refusing update from remote peer 3c747673187c42a894b270c6769ddab2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:20:18.171423 14739 consensus_queue.cc:1035] T f4d5b7b89ba34cb09e1e5dc27d26f00a P 3c747673187c42a894b270c6769ddab2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48927739d82c4f6a8274019312c386c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:20:18.171763 14734 consensus_queue.cc:1035] T f4d5b7b89ba34cb09e1e5dc27d26f00a P 3c747673187c42a894b270c6769ddab2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9fc7afd66a64bf997acf128e9f4ad28" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:20:18.193698 14253 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40942:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:20:18.193980 14253 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40942:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:20:18.197739 14253 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3b36d06442754a95908b39ef6b46ad06: Sending DeleteTablet for 3 replicas of tablet f4d5b7b89ba34cb09e1e5dc27d26f00a
I20260928 05:20:18.198763 14645 tablet_service.cc:1514] Processing DeleteTablet for tablet f4d5b7b89ba34cb09e1e5dc27d26f00a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:20:18 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41358
I20260928 05:20:18.198766 14378 tablet_service.cc:1514] Processing DeleteTablet for tablet f4d5b7b89ba34cb09e1e5dc27d26f00a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:20:18 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49742
I20260928 05:20:18.199865 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 14315
I20260928 05:20:18.204655 14512 tablet_service.cc:1514] Processing DeleteTablet for tablet f4d5b7b89ba34cb09e1e5dc27d26f00a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:20:18 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54558
I20260928 05:20:18.213320 14754 tablet_replica.cc:330] T f4d5b7b89ba34cb09e1e5dc27d26f00a P 3c747673187c42a894b270c6769ddab2: stopping tablet replica
I20260928 05:20:18.213796 14754 raft_consensus.cc:2229] T f4d5b7b89ba34cb09e1e5dc27d26f00a P 3c747673187c42a894b270c6769ddab2 [term 1 LEADER]: Raft consensus shutting down.
I20260928 05:20:18.214452 14754 raft_consensus.cc:2258] T f4d5b7b89ba34cb09e1e5dc27d26f00a P 3c747673187c42a894b270c6769ddab2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:20:18.216629 14754 ts_tablet_manager.cc:1905] T f4d5b7b89ba34cb09e1e5dc27d26f00a P 3c747673187c42a894b270c6769ddab2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:20:18.217281 14752 tablet_replica.cc:330] T f4d5b7b89ba34cb09e1e5dc27d26f00a P d9fc7afd66a64bf997acf128e9f4ad28: stopping tablet replica
I20260928 05:20:18.218185 14751 tablet_replica.cc:330] T f4d5b7b89ba34cb09e1e5dc27d26f00a P 48927739d82c4f6a8274019312c386c6: stopping tablet replica
I20260928 05:20:18.218607 14751 raft_consensus.cc:2229] T f4d5b7b89ba34cb09e1e5dc27d26f00a P 48927739d82c4f6a8274019312c386c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:20:18.219135 14751 raft_consensus.cc:2258] T f4d5b7b89ba34cb09e1e5dc27d26f00a P 48927739d82c4f6a8274019312c386c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:20:18.219089 14752 raft_consensus.cc:2229] T f4d5b7b89ba34cb09e1e5dc27d26f00a P d9fc7afd66a64bf997acf128e9f4ad28 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:20:18.220009 14752 raft_consensus.cc:2258] T f4d5b7b89ba34cb09e1e5dc27d26f00a P d9fc7afd66a64bf997acf128e9f4ad28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:20:18.221191 14751 ts_tablet_manager.cc:1905] T f4d5b7b89ba34cb09e1e5dc27d26f00a P 48927739d82c4f6a8274019312c386c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:20:18.223094 14752 ts_tablet_manager.cc:1905] T f4d5b7b89ba34cb09e1e5dc27d26f00a P d9fc7afd66a64bf997acf128e9f4ad28: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 28 05:20:18 dist-test-slave-whsn krb5kdc[14209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572815, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260928 05:20:18.227780 14751 ts_tablet_manager.cc:1918] T f4d5b7b89ba34cb09e1e5dc27d26f00a P 48927739d82c4f6a8274019312c386c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:20:18.227928 14751 log.cc:1196] T f4d5b7b89ba34cb09e1e5dc27d26f00a P 48927739d82c4f6a8274019312c386c6: Deleting WAL directory at /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-2/wal/wals/f4d5b7b89ba34cb09e1e5dc27d26f00a
I20260928 05:20:18.228399 14751 ts_tablet_manager.cc:1939] T f4d5b7b89ba34cb09e1e5dc27d26f00a P 48927739d82c4f6a8274019312c386c6: Deleting consensus metadata
I20260928 05:20:18.232539 14752 ts_tablet_manager.cc:1918] T f4d5b7b89ba34cb09e1e5dc27d26f00a P d9fc7afd66a64bf997acf128e9f4ad28: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:20:18.232741 14752 log.cc:1196] T f4d5b7b89ba34cb09e1e5dc27d26f00a P d9fc7afd66a64bf997acf128e9f4ad28: Deleting WAL directory at /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-0/wal/wals/f4d5b7b89ba34cb09e1e5dc27d26f00a
I20260928 05:20:18.233299 14752 ts_tablet_manager.cc:1939] T f4d5b7b89ba34cb09e1e5dc27d26f00a P d9fc7afd66a64bf997acf128e9f4ad28: Deleting consensus metadata
I20260928 05:20:18.235210 14240 catalog_manager.cc:4841] TS d9fc7afd66a64bf997acf128e9f4ad28 (127.0.97.1:41813): tablet f4d5b7b89ba34cb09e1e5dc27d26f00a (table test-table [id=5c238e004a4a4b999172582cd9a430b6]) successfully deleted
I20260928 05:20:18.235354 14754 ts_tablet_manager.cc:1918] T f4d5b7b89ba34cb09e1e5dc27d26f00a P 3c747673187c42a894b270c6769ddab2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:20:18.235514 14754 log.cc:1196] T f4d5b7b89ba34cb09e1e5dc27d26f00a P 3c747673187c42a894b270c6769ddab2: Deleting WAL directory at /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-1/wal/wals/f4d5b7b89ba34cb09e1e5dc27d26f00a
I20260928 05:20:18.235859 14726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:18.200820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49754 (local address 127.0.97.1:41813)
0928 05:20:18.200993 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 05:20:18.200998 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:20:18.217593 (+ 16595us) server_negotiation.cc:407] Connection header received
0928 05:20:18.217779 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:20:18.217784 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:20:18.217881 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:20:18.217982 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:20:18.219799 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:18.220941 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:18.222794 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:18.223383 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:18.223470 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:18.228486 (+ 5016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:20:18.228499 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:20:18.228516 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:20:18.228554 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:20:18.232324 (+ 3770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:18.233380 (+ 1056us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:18.233388 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:18.233391 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:18.233552 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:18.234046 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:18.234051 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:18.234055 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:18.234488 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:20:18.234764 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0928 05:20:18.235292 (+ 528us) server_negotiation.cc:299] Negotiation successful
0928 05:20:18.235480 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 05:20:18.236012 14754 ts_tablet_manager.cc:1939] T f4d5b7b89ba34cb09e1e5dc27d26f00a P 3c747673187c42a894b270c6769ddab2: Deleting consensus metadata
I20260928 05:20:18.236779 14438 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:20:18.237579 14241 catalog_manager.cc:4841] TS 3c747673187c42a894b270c6769ddab2 (127.0.97.2:37923): tablet f4d5b7b89ba34cb09e1e5dc27d26f00a (table test-table [id=5c238e004a4a4b999172582cd9a430b6]) successfully deleted
I20260928 05:20:18.237594 14238 catalog_manager.cc:4841] TS 48927739d82c4f6a8274019312c386c6 (127.0.97.3:33283): tablet f4d5b7b89ba34cb09e1e5dc27d26f00a (table test-table [id=5c238e004a4a4b999172582cd9a430b6]) successfully deleted
I20260928 05:20:18.352190 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 14315
I20260928 05:20:18.366539 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 14449
Sep 28 05:20:18 dist-test-slave-whsn krb5kdc[14209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572815, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260928 05:20:18.382853 14727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:18.367521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54566 (local address 127.0.97.2:37923)
0928 05:20:18.367703 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 05:20:18.367709 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:20:18.367726 (+ 17us) server_negotiation.cc:407] Connection header received
0928 05:20:18.367787 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:20:18.367790 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:20:18.367852 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:20:18.367931 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:20:18.369476 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:18.370243 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:18.372113 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:18.372546 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:18.372638 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:18.377315 (+ 4677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:20:18.377326 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:20:18.377340 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:20:18.377384 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:20:18.380065 (+ 2681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:18.380728 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:18.380735 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:18.380737 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:18.381103 (+ 366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:18.381436 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:18.381441 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:18.381444 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:18.381757 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:20:18.382099 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0928 05:20:18.382323 (+ 224us) server_negotiation.cc:299] Negotiation successful
0928 05:20:18.382522 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 05:20:18.383476 14572 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:20:18.492558 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 14449
I20260928 05:20:18.508633 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 14582
Sep 28 05:20:18 dist-test-slave-whsn krb5kdc[14209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572815, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260928 05:20:18.522042 14723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:18.509316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41370 (local address 127.0.97.3:33283)
0928 05:20:18.509462 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 05:20:18.509466 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:20:18.509508 (+ 42us) server_negotiation.cc:407] Connection header received
0928 05:20:18.509671 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:20:18.509675 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:20:18.509736 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:20:18.509814 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:20:18.510944 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:18.511628 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:18.512777 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:18.513278 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:18.513362 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:18.517519 (+ 4157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:20:18.517530 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:20:18.517545 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:20:18.517589 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:20:18.519821 (+ 2232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:18.520400 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:18.520408 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:18.520410 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:18.520555 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:18.520822 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:18.520824 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:18.520826 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:18.521222 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:20:18.521421 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0928 05:20:18.521651 (+ 230us) server_negotiation.cc:299] Negotiation successful
0928 05:20:18.521799 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 05:20:18.522608 14705 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:20:18.608014 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 14582
I20260928 05:20:18.620879 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 14222
I20260928 05:20:18.621923 14284 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:20:18.729732 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 14222
I20260928 05:20:18.746475 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38873
--webserver_interface=127.0.97.62
--webserver_port=40133
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38873
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:20:19.048439 14759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:20:19.048722 14759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:20:19.048863 14759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:20:19.055927 14759 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:20:19.056028 14759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:20:19.056097 14759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:20:19.056170 14759 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:20:19.056228 14759 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:20:19.065862 14759 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38873
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=40133
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:20:19.067087 14759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:20:19.068321 14759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:20:19.088665 14765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:20:19.088641 14768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:20:19.088711 14766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:20:19.089402 14759 server_base.cc:1034] running on GCE node
I20260928 05:20:19.090229 14759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:20:19.091735 14759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:20:19.092941 14759 hybrid_clock.cc:648] HybridClock initialized: now 1790572819092884 us; error 43 us; skew 500 ppm
I20260928 05:20:19.093320 14759 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:20:19 dist-test-slave-whsn krb5kdc[14209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572819, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:20:19.096607 14759 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 05:20:19.098145 14759 webserver.cc:460] Webserver started at http://127.0.97.62:40133/ using document root <none> and password file <none>
I20260928 05:20:19.098824 14759 fs_manager.cc:359] Metadata directory not provided
I20260928 05:20:19.098948 14759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:20:19.104457 14759 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260928 05:20:19.107225 14775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:20:19.108455 14759 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 05:20:19.108608 14759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/master-0/wal
uuid: "3b36d06442754a95908b39ef6b46ad06"
format_stamp: "Formatted at 2026-09-28 05:20:16 on dist-test-slave-whsn"
server_key: "e98dabea40fe88d3c480d40e6b9ff037"
server_key_iv: "77733407db60baee2278fc1927cf2899"
server_key_version: "encryptionkey@0"
I20260928 05:20:19.109639 14759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:20:19.122848 14759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:20:19.126725 14759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:20:19.126955 14759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:20:19.143744 14759 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38873
I20260928 05:20:19.143774 14826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38873 every 8 connection(s)
I20260928 05:20:19.145171 14759 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:20:19.151887 14827 sys_catalog.cc:263] Verifying existing consensus state
I20260928 05:20:19.154809 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 14759
I20260928 05:20:19.155226 14827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b36d06442754a95908b39ef6b46ad06: Bootstrap starting.
I20260928 05:20:19.155728 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:41813
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:38873
--webserver_port=42701
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 05:20:19.175940 14827 log.cc:824] T 00000000000000000000000000000000 P 3b36d06442754a95908b39ef6b46ad06: Log is configured to *not* fsync() on all Append() calls
I20260928 05:20:19.187412 14827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b36d06442754a95908b39ef6b46ad06: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 05:20:19.187879 14827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b36d06442754a95908b39ef6b46ad06: Bootstrap complete.
I20260928 05:20:19.196513 14827 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b36d06442754a95908b39ef6b46ad06 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b36d06442754a95908b39ef6b46ad06" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38873 } }
I20260928 05:20:19.197332 14827 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b36d06442754a95908b39ef6b46ad06 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b36d06442754a95908b39ef6b46ad06, State: Initialized, Role: FOLLOWER
I20260928 05:20:19.197877 14827 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b36d06442754a95908b39ef6b46ad06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b36d06442754a95908b39ef6b46ad06" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38873 } }
I20260928 05:20:19.198099 14827 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b36d06442754a95908b39ef6b46ad06 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:20:19.198202 14827 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b36d06442754a95908b39ef6b46ad06 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:20:19.198357 14827 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b36d06442754a95908b39ef6b46ad06 [term 1 FOLLOWER]: Advancing to term 2
I20260928 05:20:19.201973 14827 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b36d06442754a95908b39ef6b46ad06 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b36d06442754a95908b39ef6b46ad06" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38873 } }
I20260928 05:20:19.202426 14827 leader_election.cc:302] T 00000000000000000000000000000000 P 3b36d06442754a95908b39ef6b46ad06 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b36d06442754a95908b39ef6b46ad06; no voters:
I20260928 05:20:19.202978 14827 leader_election.cc:288] T 00000000000000000000000000000000 P 3b36d06442754a95908b39ef6b46ad06 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 05:20:19.203176 14831 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b36d06442754a95908b39ef6b46ad06 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 05:20:19.204154 14831 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b36d06442754a95908b39ef6b46ad06 [term 2 LEADER]: Becoming Leader. State: Replica: 3b36d06442754a95908b39ef6b46ad06, State: Running, Role: LEADER
I20260928 05:20:19.204742 14831 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b36d06442754a95908b39ef6b46ad06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b36d06442754a95908b39ef6b46ad06" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38873 } }
I20260928 05:20:19.205168 14827 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b36d06442754a95908b39ef6b46ad06 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:20:19.207068 14832 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b36d06442754a95908b39ef6b46ad06 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "3b36d06442754a95908b39ef6b46ad06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b36d06442754a95908b39ef6b46ad06" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38873 } } }
I20260928 05:20:19.207418 14832 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b36d06442754a95908b39ef6b46ad06 [sys.catalog]: This master's current role is: LEADER
I20260928 05:20:19.208745 14833 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b36d06442754a95908b39ef6b46ad06 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b36d06442754a95908b39ef6b46ad06. Latest consensus state: current_term: 2 leader_uuid: "3b36d06442754a95908b39ef6b46ad06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b36d06442754a95908b39ef6b46ad06" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38873 } } }
I20260928 05:20:19.208978 14836 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:20:19.209098 14833 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b36d06442754a95908b39ef6b46ad06 [sys.catalog]: This master's current role is: LEADER
I20260928 05:20:19.217190 14836 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:20:19.218677 14836 catalog_manager.cc:1259] Loaded cluster ID: 300d65fd8cab49eda278356e8e700e03
I20260928 05:20:19.218833 14836 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:20:19.222074 14836 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260928 05:20:19.240605 14836 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260928 05:20:19.256608 14836 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:20:19.258322 14836 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3b36d06442754a95908b39ef6b46ad06: Loaded TSK: 0
I20260928 05:20:19.259754 14836 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 05:20:19.481541 14829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:20:19.481799 14829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:20:19.481913 14829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:20:19.489006 14829 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:20:19.489096 14829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:20:19.489230 14829 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 05:20:19.498499 14829 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:41813
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=42701
--tserver_master_addrs=127.0.97.62:38873
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:20:19.499639 14829 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:20:19.500872 14829 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:20:19.514971 14854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:20:19.514897 14855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:20:19.515790 14857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:20:19.515939 14829 server_base.cc:1034] running on GCE node
I20260928 05:20:19.516536 14829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:20:19.517431 14829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:20:19.518625 14829 hybrid_clock.cc:648] HybridClock initialized: now 1790572819518562 us; error 52 us; skew 500 ppm
I20260928 05:20:19.518906 14829 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:20:19 dist-test-slave-whsn krb5kdc[14209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572819, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:20:19.521807 14829 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260928 05:20:19.523154 14829 webserver.cc:460] Webserver started at http://127.0.97.1:42701/ using document root <none> and password file <none>
I20260928 05:20:19.523768 14829 fs_manager.cc:359] Metadata directory not provided
I20260928 05:20:19.523880 14829 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:20:19.529016 14829 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260928 05:20:19.531545 14864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:20:19.532825 14829 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 05:20:19.532992 14829 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-0/wal
uuid: "d9fc7afd66a64bf997acf128e9f4ad28"
format_stamp: "Formatted at 2026-09-28 05:20:16 on dist-test-slave-whsn"
server_key: "abfff00c90c1ec802b746e533217ed74"
server_key_iv: "6bdbfa0c04b72d97b3fe600767c65655"
server_key_version: "encryptionkey@0"
I20260928 05:20:19.533871 14829 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:20:19.552380 14829 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:20:19.556119 14829 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:20:19.556337 14829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:20:19.557271 14829 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:20:19.558463 14829 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:20:19.558549 14829 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:20:19.558658 14829 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:20:19.558729 14829 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:20:19.585681 14829 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41813
I20260928 05:20:19.585744 14976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41813 every 8 connection(s)
I20260928 05:20:19.586980 14829 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-0/data/info.pb
I20260928 05:20:19.588097 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 14829
I20260928 05:20:19.588784 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:37923
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:38873
--webserver_port=37261
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 05:20:19 dist-test-slave-whsn krb5kdc[14209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572819, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:20:19.608894 14979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:19.589836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45011 (local address 127.0.97.62:38873)
0928 05:20:19.590520 (+ 684us) server_negotiation.cc:206] Beginning negotiation
0928 05:20:19.590535 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 05:20:19.590999 (+ 464us) server_negotiation.cc:407] Connection header received
0928 05:20:19.592978 (+ 1979us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:20:19.592991 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:20:19.593308 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:20:19.593628 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:20:19.595056 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:19.596290 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:19.597784 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:19.598401 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:19.598563 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:19.601961 (+ 3398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:20:19.601968 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:20:19.601980 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:20:19.602020 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:20:19.604658 (+ 2638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:19.605537 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:19.605546 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:19.605554 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:19.605748 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:19.606194 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:19.606207 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:19.606209 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:19.606713 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:20:19.606860 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 05:20:19.607693 (+ 833us) server_negotiation.cc:299] Negotiation successful
0928 05:20:19.607987 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":339,"threads_started":1}
I20260928 05:20:19.609338 14980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:19.590146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38873 (local address 127.0.97.1:45011)
0928 05:20:19.590845 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:19.590881 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 05:20:19.592441 (+ 1560us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:20:19.593796 (+ 1355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:20:19.593807 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:20:19.594346 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:20:19.594848 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:20:19.594865 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:20:19.596439 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:20:19.596443 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:20:19.597656 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:20:19.597668 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:20:19.598655 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:20:19.598664 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:20:19.598887 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:19.599455 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:20:19.599480 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:20:19.601709 (+ 2229us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:20:19.604799 (+ 3090us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:20:19.604808 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:20:19.604823 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:20:19.605410 (+ 587us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:20:19.605851 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:20:19.605855 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:20:19.605859 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:20:19.606098 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:20:19.606830 (+ 732us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:20:19.606838 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:20:19.607246 (+ 408us) client_negotiation.cc:769] Sending connection context
0928 05:20:19.607548 (+ 302us) client_negotiation.cc:240] Negotiation successful
0928 05:20:19.608065 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":278,"threads_started":1}
I20260928 05:20:19.611961 14977 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38873
I20260928 05:20:19.612254 14977 heartbeater.cc:464] Registering TS with master...
I20260928 05:20:19.613116 14977 heartbeater.cc:511] Master 127.0.97.62:38873 requested a full tablet report, sending...
I20260928 05:20:19.615171 14792 ts_manager.cc:194] Registered new tserver with Master: d9fc7afd66a64bf997acf128e9f4ad28 (127.0.97.1:41813)
I20260928 05:20:19.616496 14792 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:45011
W20260928 05:20:19.894801 14981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:20:19.895085 14981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:20:19.895228 14981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:20:19.902360 14981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:20:19.902498 14981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:20:19.902658 14981 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 05:20:19.912415 14981 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:37923
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=37261
--tserver_master_addrs=127.0.97.62:38873
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:20:19.913628 14981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:20:19.914842 14981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:20:19.926321 14986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:20:19.926352 14989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:20:19.926435 14987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:20:19.927328 14981 server_base.cc:1034] running on GCE node
I20260928 05:20:19.927837 14981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:20:19.928658 14981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:20:19.929865 14981 hybrid_clock.cc:648] HybridClock initialized: now 1790572819929797 us; error 52 us; skew 500 ppm
I20260928 05:20:19.930169 14981 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:20:19 dist-test-slave-whsn krb5kdc[14209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572819, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:20:19.932950 14981 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260928 05:20:19.934192 14981 webserver.cc:460] Webserver started at http://127.0.97.2:37261/ using document root <none> and password file <none>
I20260928 05:20:19.934772 14981 fs_manager.cc:359] Metadata directory not provided
I20260928 05:20:19.934890 14981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:20:19.939913 14981 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 05:20:19.942363 14996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:20:19.943545 14981 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260928 05:20:19.943681 14981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-1/wal
uuid: "3c747673187c42a894b270c6769ddab2"
format_stamp: "Formatted at 2026-09-28 05:20:17 on dist-test-slave-whsn"
server_key: "b3f72ace898d07ce94f3ff18a606b377"
server_key_iv: "dc2d72eee7b3063c3403165fe5fbdcf1"
server_key_version: "encryptionkey@0"
I20260928 05:20:19.944589 14981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:20:19.967495 14981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:20:19.971240 14981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:20:19.971460 14981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:20:19.972338 14981 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:20:19.973575 14981 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:20:19.973675 14981 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:20:19.973774 14981 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:20:19.973845 14981 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:20:20.002235 14981 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37923
I20260928 05:20:20.002295 15108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37923 every 8 connection(s)
I20260928 05:20:20.003594 14981 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-1/data/info.pb
I20260928 05:20:20.007659 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 14981
I20260928 05:20:20.008359 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:33283
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:38873
--webserver_port=41963
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 05:20:20 dist-test-slave-whsn krb5kdc[14209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572819, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:20:20.023131 14979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:20.005971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51347 (local address 127.0.97.62:38873)
0928 05:20:20.006188 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 05:20:20.006193 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:20:20.007268 (+ 1075us) server_negotiation.cc:407] Connection header received
0928 05:20:20.009665 (+ 2397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:20:20.009669 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:20:20.009739 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:20:20.009819 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:20:20.011571 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:20.012266 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:20.014287 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:20.014673 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:20.014762 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:20.018099 (+ 3337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:20:20.018104 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:20:20.018107 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:20:20.018140 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:20:20.020356 (+ 2216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:20.021052 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:20.021057 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:20.021060 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:20.021207 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:20.021571 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:20.021576 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:20.021579 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:20.021879 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:20:20.022008 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0928 05:20:20.022575 (+ 567us) server_negotiation.cc:299] Negotiation successful
0928 05:20:20.022800 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 05:20:20.023794 15111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:20.006462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38873 (local address 127.0.97.2:51347)
0928 05:20:20.007111 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:20.007152 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 05:20:20.009382 (+ 2230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:20:20.010061 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:20:20.010071 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:20:20.010712 (+ 641us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:20:20.011427 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:20:20.011450 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:20:20.012404 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:20:20.012410 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:20:20.014162 (+ 1752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:20:20.014176 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:20:20.014861 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:20:20.014867 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:20:20.015112 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:20.015734 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:20:20.015760 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:20:20.017891 (+ 2131us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:20:20.020477 (+ 2586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:20:20.020485 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:20:20.020500 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:20:20.020922 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:20:20.021304 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:20:20.021307 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:20:20.021310 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:20:20.021470 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:20:20.021979 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:20:20.021984 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:20:20.022250 (+ 266us) client_negotiation.cc:769] Sending connection context
0928 05:20:20.022449 (+ 199us) client_negotiation.cc:240] Negotiation successful
0928 05:20:20.022791 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":248,"threads_started":1}
I20260928 05:20:20.025261 15109 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38873
I20260928 05:20:20.025520 15109 heartbeater.cc:464] Registering TS with master...
I20260928 05:20:20.026141 15109 heartbeater.cc:511] Master 127.0.97.62:38873 requested a full tablet report, sending...
I20260928 05:20:20.027567 14792 ts_manager.cc:194] Registered new tserver with Master: 3c747673187c42a894b270c6769ddab2 (127.0.97.2:37923)
I20260928 05:20:20.028645 14792 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:51347
W20260928 05:20:20.311931 15112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:20:20.312181 15112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:20:20.312273 15112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:20:20.319263 15112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:20:20.319356 15112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:20:20.319470 15112 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 05:20:20.328626 15112 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:33283
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=41963
--tserver_master_addrs=127.0.97.62:38873
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:20:20.329761 15112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:20:20.330945 15112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:20:20.342672 15117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:20:20.342654 15118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:20:20.342713 15120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:20:20.343390 15112 server_base.cc:1034] running on GCE node
I20260928 05:20:20.343887 15112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:20:20.344702 15112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:20:20.345897 15112 hybrid_clock.cc:648] HybridClock initialized: now 1790572820345842 us; error 51 us; skew 500 ppm
I20260928 05:20:20.346149 15112 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:20:20 dist-test-slave-whsn krb5kdc[14209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572820, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:20:20.352561 15112 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260928 05:20:20.353904 15112 webserver.cc:460] Webserver started at http://127.0.97.3:41963/ using document root <none> and password file <none>
I20260928 05:20:20.354475 15112 fs_manager.cc:359] Metadata directory not provided
I20260928 05:20:20.354578 15112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:20:20.359414 15112 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260928 05:20:20.361874 15127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:20:20.363080 15112 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 05:20:20.363214 15112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-2/wal
uuid: "48927739d82c4f6a8274019312c386c6"
format_stamp: "Formatted at 2026-09-28 05:20:17 on dist-test-slave-whsn"
server_key: "a9c2679720605cdf6dedaff6d97079d3"
server_key_iv: "87e381d6de55e57ee2203c2d2cd41123"
server_key_version: "encryptionkey@0"
I20260928 05:20:20.364061 15112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:20:20.382040 15112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:20:20.385938 15112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:20:20.386154 15112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:20:20.387019 15112 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:20:20.388202 15112 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:20:20.388299 15112 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:20:20.388387 15112 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:20:20.388440 15112 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:20:20.417483 15112 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33283
I20260928 05:20:20.417549 15239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33283 every 8 connection(s)
I20260928 05:20:20.418996 15112 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-2/data/info.pb
I20260928 05:20:20.419405 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 15112
Sep 28 05:20:20 dist-test-slave-whsn krb5kdc[14209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572820, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:20:20.435348 14979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:20.420344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40958 (local address 127.0.97.62:38873)
0928 05:20:20.420533 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 05:20:20.420539 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:20:20.420978 (+ 439us) server_negotiation.cc:407] Connection header received
0928 05:20:20.421162 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:20:20.421166 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:20:20.421232 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:20:20.421319 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:20:20.423218 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:20.424288 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:20.425623 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:20.426234 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:20.426360 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:20.429551 (+ 3191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:20:20.429558 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:20:20.429563 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:20:20.429608 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:20:20.432841 (+ 3233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:20.433518 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:20.433524 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:20.433527 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:20.433656 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:20.433998 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:20.434003 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:20.434006 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:20.434271 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:20:20.434518 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0928 05:20:20.434832 (+ 314us) server_negotiation.cc:299] Negotiation successful
0928 05:20:20.435036 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 05:20:20.440488 15243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:20.421486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51015 (local address 127.0.97.62:38873)
0928 05:20:20.421874 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0928 05:20:20.421878 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:20:20.422966 (+ 1088us) server_negotiation.cc:407] Connection header received
0928 05:20:20.425121 (+ 2155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:20:20.425126 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:20:20.425197 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:20:20.425290 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:20:20.427047 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:20.428100 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:20.429767 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:20.430370 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:20.430498 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:20.434161 (+ 3663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:20:20.434166 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:20:20.434169 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:20:20.434200 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:20:20.437317 (+ 3117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:20.438189 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:20.438195 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:20.438199 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:20.438317 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:20.438741 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:20.438747 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:20.438751 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:20.439048 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:20:20.439180 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 05:20:20.439848 (+ 668us) server_negotiation.cc:299] Negotiation successful
0928 05:20:20.440045 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":215,"threads_started":1}
I20260928 05:20:20.441344 15244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:20.422133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38873 (local address 127.0.97.3:51015)
0928 05:20:20.422830 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:20.422871 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 05:20:20.424896 (+ 2025us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:20:20.425524 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:20:20.425532 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:20:20.426176 (+ 644us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:20:20.426855 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:20:20.426878 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:20:20.428266 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:20:20.428270 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:20:20.429619 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:20:20.429633 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:20:20.430578 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:20:20.430585 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:20:20.430877 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:20.431484 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:20:20.431511 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:20:20.433928 (+ 2417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:20:20.437450 (+ 3522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:20:20.437459 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:20:20.437473 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:20:20.438045 (+ 572us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:20:20.438423 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:20:20.438429 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:20:20.438433 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:20:20.438622 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:20:20.439167 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:20:20.439175 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:20:20.439507 (+ 332us) client_negotiation.cc:769] Sending connection context
0928 05:20:20.439743 (+ 236us) client_negotiation.cc:240] Negotiation successful
0928 05:20:20.440207 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":289,"threads_started":1}
I20260928 05:20:20.442559 15240 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38873
I20260928 05:20:20.442818 15240 heartbeater.cc:464] Registering TS with master...
I20260928 05:20:20.443444 15240 heartbeater.cc:511] Master 127.0.97.62:38873 requested a full tablet report, sending...
I20260928 05:20:20.444819 14792 ts_manager.cc:194] Registered new tserver with Master: 48927739d82c4f6a8274019312c386c6 (127.0.97.3:33283)
I20260928 05:20:20.445700 14792 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:51015
I20260928 05:20:20.449828 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 05:20:20.456144 15243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:20.450700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40970 (local address 127.0.97.62:38873)
0928 05:20:20.450840 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 05:20:20.450844 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:20:20.451300 (+ 456us) server_negotiation.cc:407] Connection header received
0928 05:20:20.451462 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:20:20.451465 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:20:20.451610 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:20:20.451722 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:20:20.452112 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:20.452772 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:20.454293 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:20.454715 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:20.454803 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:20.455259 (+ 456us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:20:20.455718 (+ 459us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:20:20.455817 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 05:20:20.455904 (+ 87us) server_negotiation.cc:299] Negotiation successful
0928 05:20:20.455961 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 05:20:20.457039 14792 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:40970:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 05:20:20.464591 14792 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 05:20:20.506110 15250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:20.477817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41813 (local address 127.0.0.1:49770)
0928 05:20:20.495915 (+ 18098us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:20.495961 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 05:20:20.496278 (+ 317us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:20:20.497443 (+ 1165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:20:20.497452 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:20:20.497498 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:20:20.497930 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:20:20.497946 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:20:20.500126 (+ 2180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:20:20.500133 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:20:20.502653 (+ 2520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:20:20.502672 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:20:20.505005 (+ 2333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:20:20.505013 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:20:20.505229 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:20.505291 (+ 62us) client_negotiation.cc:769] Sending connection context
0928 05:20:20.505453 (+ 162us) client_negotiation.cc:240] Negotiation successful
0928 05:20:20.505669 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17834,"thread_start_us":17685,"threads_started":1}
I20260928 05:20:20.507170 15251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:20.477900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49770 (local address 127.0.97.1:41813)
0928 05:20:20.495094 (+ 17194us) server_negotiation.cc:206] Beginning negotiation
0928 05:20:20.495107 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 05:20:20.496101 (+ 994us) server_negotiation.cc:407] Connection header received
0928 05:20:20.496554 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:20:20.496565 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:20:20.496898 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:20:20.497139 (+ 241us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:20:20.498196 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:20.499929 (+ 1733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:20.502861 (+ 2932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:20.504419 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:20.504541 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:20.504767 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0928 05:20:20.506606 (+ 1839us) server_negotiation.cc:299] Negotiation successful
0928 05:20:20.506789 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16961,"thread_start_us":17214,"threads_started":1}
I20260928 05:20:20.511340 14912 tablet_service.cc:1467] Processing CreateTablet for tablet 72ba574bbee64b239d3c406e5fbde300 (DEFAULT_TABLE table=test-table [id=7507f5c88bf84b1198a15700cac9a7cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:20:20.512820 14912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72ba574bbee64b239d3c406e5fbde300. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:20:20.523324 15252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:20.479148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37923 (local address 127.0.0.1:54576)
0928 05:20:20.507625 (+ 28477us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:20.507644 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 05:20:20.508951 (+ 1307us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:20:20.510967 (+ 2016us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:20:20.510972 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:20:20.510996 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:20:20.511295 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:20:20.511306 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:20:20.515068 (+ 3762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:20:20.515072 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:20:20.517660 (+ 2588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:20:20.517675 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:20:20.522556 (+ 4881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:20:20.522562 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:20:20.522739 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:20.522772 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 05:20:20.522867 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 05:20:20.522996 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28369,"thread_start_us":26827,"threads_started":1}
I20260928 05:20:20.523535 15255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:20.479769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54576 (local address 127.0.97.2:37923)
0928 05:20:20.505063 (+ 25294us) server_negotiation.cc:206] Beginning negotiation
0928 05:20:20.505071 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:20:20.507755 (+ 2684us) server_negotiation.cc:407] Connection header received
0928 05:20:20.509170 (+ 1415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:20:20.509178 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:20:20.509400 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:20:20.509542 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:20:20.512879 (+ 3337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:20.514903 (+ 2024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:20.517833 (+ 2930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:20.520254 (+ 2421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:20.520347 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:20.520564 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0928 05:20:20.522969 (+ 2405us) server_negotiation.cc:299] Negotiation successful
0928 05:20:20.523156 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25119,"thread_start_us":25322,"threads_started":1}
I20260928 05:20:20.525290 15253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:20.478607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33283 (local address 127.0.0.1:41384)
0928 05:20:20.505754 (+ 27147us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:20.505774 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 05:20:20.505900 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:20:20.508847 (+ 2947us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:20:20.508878 (+ 31us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:20:20.508910 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:20:20.509693 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:20:20.509711 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:20:20.517518 (+ 7807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:20:20.517524 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:20:20.519608 (+ 2084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:20:20.519626 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:20:20.523626 (+ 4000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:20:20.524498 (+ 872us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:20:20.524671 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:20.524703 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 05:20:20.524796 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 05:20:20.524937 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27015,"thread_start_us":28783,"threads_started":1}
I20260928 05:20:20.526033 15254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:20.479167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41384 (local address 127.0.97.3:33283)
0928 05:20:20.508261 (+ 29094us) server_negotiation.cc:206] Beginning negotiation
0928 05:20:20.508269 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:20:20.508299 (+ 30us) server_negotiation.cc:407] Connection header received
0928 05:20:20.508393 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:20:20.508401 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:20:20.508648 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:20:20.508830 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:20:20.511688 (+ 2858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:20.517339 (+ 5651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:20.520652 (+ 3313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:20.522031 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:20.522167 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:20.522372 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0928 05:20:20.525572 (+ 3200us) server_negotiation.cc:299] Negotiation successful
0928 05:20:20.525725 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28937,"thread_start_us":28773,"threads_started":1}
I20260928 05:20:20.526867 15044 tablet_service.cc:1467] Processing CreateTablet for tablet 72ba574bbee64b239d3c406e5fbde300 (DEFAULT_TABLE table=test-table [id=7507f5c88bf84b1198a15700cac9a7cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:20:20.527859 15044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72ba574bbee64b239d3c406e5fbde300. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:20:20.532271 15175 tablet_service.cc:1467] Processing CreateTablet for tablet 72ba574bbee64b239d3c406e5fbde300 (DEFAULT_TABLE table=test-table [id=7507f5c88bf84b1198a15700cac9a7cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:20:20.533545 15175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72ba574bbee64b239d3c406e5fbde300. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:20:20.533524 15256 tablet_bootstrap.cc:492] T 72ba574bbee64b239d3c406e5fbde300 P d9fc7afd66a64bf997acf128e9f4ad28: Bootstrap starting.
I20260928 05:20:20.537348 15256 tablet_bootstrap.cc:654] T 72ba574bbee64b239d3c406e5fbde300 P d9fc7afd66a64bf997acf128e9f4ad28: Neither blocks nor log segments found. Creating new log.
I20260928 05:20:20.538756 15256 log.cc:824] T 72ba574bbee64b239d3c406e5fbde300 P d9fc7afd66a64bf997acf128e9f4ad28: Log is configured to *not* fsync() on all Append() calls
I20260928 05:20:20.545639 15258 tablet_bootstrap.cc:492] T 72ba574bbee64b239d3c406e5fbde300 P 3c747673187c42a894b270c6769ddab2: Bootstrap starting.
I20260928 05:20:20.546741 15256 tablet_bootstrap.cc:492] T 72ba574bbee64b239d3c406e5fbde300 P d9fc7afd66a64bf997acf128e9f4ad28: No bootstrap required, opened a new log
I20260928 05:20:20.547017 15256 ts_tablet_manager.cc:1397] T 72ba574bbee64b239d3c406e5fbde300 P d9fc7afd66a64bf997acf128e9f4ad28: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.007s
I20260928 05:20:20.548432 15258 tablet_bootstrap.cc:654] T 72ba574bbee64b239d3c406e5fbde300 P 3c747673187c42a894b270c6769ddab2: Neither blocks nor log segments found. Creating new log.
I20260928 05:20:20.549480 15258 log.cc:824] T 72ba574bbee64b239d3c406e5fbde300 P 3c747673187c42a894b270c6769ddab2: Log is configured to *not* fsync() on all Append() calls
I20260928 05:20:20.554824 15259 tablet_bootstrap.cc:492] T 72ba574bbee64b239d3c406e5fbde300 P 48927739d82c4f6a8274019312c386c6: Bootstrap starting.
I20260928 05:20:20.557690 15258 tablet_bootstrap.cc:492] T 72ba574bbee64b239d3c406e5fbde300 P 3c747673187c42a894b270c6769ddab2: No bootstrap required, opened a new log
I20260928 05:20:20.557621 15256 raft_consensus.cc:348] T 72ba574bbee64b239d3c406e5fbde300 P d9fc7afd66a64bf997acf128e9f4ad28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9fc7afd66a64bf997acf128e9f4ad28" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41813 } } peers { permanent_uuid: "48927739d82c4f6a8274019312c386c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33283 } } peers { permanent_uuid: "3c747673187c42a894b270c6769ddab2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37923 } }
I20260928 05:20:20.557955 15258 ts_tablet_manager.cc:1397] T 72ba574bbee64b239d3c406e5fbde300 P 3c747673187c42a894b270c6769ddab2: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.001s
I20260928 05:20:20.558010 15256 raft_consensus.cc:374] T 72ba574bbee64b239d3c406e5fbde300 P d9fc7afd66a64bf997acf128e9f4ad28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:20:20.558120 15256 raft_consensus.cc:729] T 72ba574bbee64b239d3c406e5fbde300 P d9fc7afd66a64bf997acf128e9f4ad28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9fc7afd66a64bf997acf128e9f4ad28, State: Initialized, Role: FOLLOWER
I20260928 05:20:20.558647 15259 tablet_bootstrap.cc:654] T 72ba574bbee64b239d3c406e5fbde300 P 48927739d82c4f6a8274019312c386c6: Neither blocks nor log segments found. Creating new log.
I20260928 05:20:20.558705 15256 consensus_queue.cc:260] T 72ba574bbee64b239d3c406e5fbde300 P d9fc7afd66a64bf997acf128e9f4ad28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9fc7afd66a64bf997acf128e9f4ad28" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41813 } } peers { permanent_uuid: "48927739d82c4f6a8274019312c386c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33283 } } peers { permanent_uuid: "3c747673187c42a894b270c6769ddab2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37923 } }
I20260928 05:20:20.559823 14977 heartbeater.cc:503] Master 127.0.97.62:38873 was elected leader, sending a full tablet report...
I20260928 05:20:20.559818 15259 log.cc:824] T 72ba574bbee64b239d3c406e5fbde300 P 48927739d82c4f6a8274019312c386c6: Log is configured to *not* fsync() on all Append() calls
I20260928 05:20:20.560300 15256 ts_tablet_manager.cc:1428] T 72ba574bbee64b239d3c406e5fbde300 P d9fc7afd66a64bf997acf128e9f4ad28: Time spent starting tablet: real 0.013s user 0.010s sys 0.004s
I20260928 05:20:20.562729 15259 tablet_bootstrap.cc:492] T 72ba574bbee64b239d3c406e5fbde300 P 48927739d82c4f6a8274019312c386c6: No bootstrap required, opened a new log
I20260928 05:20:20.562979 15259 ts_tablet_manager.cc:1397] T 72ba574bbee64b239d3c406e5fbde300 P 48927739d82c4f6a8274019312c386c6: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260928 05:20:20.568742 15258 raft_consensus.cc:348] T 72ba574bbee64b239d3c406e5fbde300 P 3c747673187c42a894b270c6769ddab2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9fc7afd66a64bf997acf128e9f4ad28" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41813 } } peers { permanent_uuid: "48927739d82c4f6a8274019312c386c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33283 } } peers { permanent_uuid: "3c747673187c42a894b270c6769ddab2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37923 } }
I20260928 05:20:20.569195 15258 raft_consensus.cc:374] T 72ba574bbee64b239d3c406e5fbde300 P 3c747673187c42a894b270c6769ddab2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:20:20.569288 15258 raft_consensus.cc:729] T 72ba574bbee64b239d3c406e5fbde300 P 3c747673187c42a894b270c6769ddab2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c747673187c42a894b270c6769ddab2, State: Initialized, Role: FOLLOWER
I20260928 05:20:20.569873 15258 consensus_queue.cc:260] T 72ba574bbee64b239d3c406e5fbde300 P 3c747673187c42a894b270c6769ddab2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9fc7afd66a64bf997acf128e9f4ad28" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41813 } } peers { permanent_uuid: "48927739d82c4f6a8274019312c386c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33283 } } peers { permanent_uuid: "3c747673187c42a894b270c6769ddab2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37923 } }
I20260928 05:20:20.570863 15109 heartbeater.cc:503] Master 127.0.97.62:38873 was elected leader, sending a full tablet report...
I20260928 05:20:20.571519 15258 ts_tablet_manager.cc:1428] T 72ba574bbee64b239d3c406e5fbde300 P 3c747673187c42a894b270c6769ddab2: Time spent starting tablet: real 0.013s user 0.010s sys 0.003s
I20260928 05:20:20.572546 15259 raft_consensus.cc:348] T 72ba574bbee64b239d3c406e5fbde300 P 48927739d82c4f6a8274019312c386c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9fc7afd66a64bf997acf128e9f4ad28" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41813 } } peers { permanent_uuid: "48927739d82c4f6a8274019312c386c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33283 } } peers { permanent_uuid: "3c747673187c42a894b270c6769ddab2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37923 } }
I20260928 05:20:20.572914 15259 raft_consensus.cc:374] T 72ba574bbee64b239d3c406e5fbde300 P 48927739d82c4f6a8274019312c386c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:20:20.573009 15259 raft_consensus.cc:729] T 72ba574bbee64b239d3c406e5fbde300 P 48927739d82c4f6a8274019312c386c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48927739d82c4f6a8274019312c386c6, State: Initialized, Role: FOLLOWER
I20260928 05:20:20.573460 15259 consensus_queue.cc:260] T 72ba574bbee64b239d3c406e5fbde300 P 48927739d82c4f6a8274019312c386c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9fc7afd66a64bf997acf128e9f4ad28" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41813 } } peers { permanent_uuid: "48927739d82c4f6a8274019312c386c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33283 } } peers { permanent_uuid: "3c747673187c42a894b270c6769ddab2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37923 } }
I20260928 05:20:20.574309 15240 heartbeater.cc:503] Master 127.0.97.62:38873 was elected leader, sending a full tablet report...
I20260928 05:20:20.574913 15259 ts_tablet_manager.cc:1428] T 72ba574bbee64b239d3c406e5fbde300 P 48927739d82c4f6a8274019312c386c6: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
W20260928 05:20:20.589099 14978 tablet.cc:2369] T 72ba574bbee64b239d3c406e5fbde300 P d9fc7afd66a64bf997acf128e9f4ad28: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 05:20:20.670791 15241 tablet.cc:2369] T 72ba574bbee64b239d3c406e5fbde300 P 48927739d82c4f6a8274019312c386c6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 05:20:20.755332 15110 tablet.cc:2369] T 72ba574bbee64b239d3c406e5fbde300 P 3c747673187c42a894b270c6769ddab2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 05:20:20.785604 15263 raft_consensus.cc:482] T 72ba574bbee64b239d3c406e5fbde300 P 3c747673187c42a894b270c6769ddab2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:20:20.785920 15263 raft_consensus.cc:504] T 72ba574bbee64b239d3c406e5fbde300 P 3c747673187c42a894b270c6769ddab2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9fc7afd66a64bf997acf128e9f4ad28" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41813 } } peers { permanent_uuid: "48927739d82c4f6a8274019312c386c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33283 } } peers { permanent_uuid: "3c747673187c42a894b270c6769ddab2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37923 } }
I20260928 05:20:20.787814 15263 leader_election.cc:288] T 72ba574bbee64b239d3c406e5fbde300 P 3c747673187c42a894b270c6769ddab2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9fc7afd66a64bf997acf128e9f4ad28 (127.0.97.1:41813), 48927739d82c4f6a8274019312c386c6 (127.0.97.3:33283)
I20260928 05:20:20.794319 15254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:20.788579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37257 (local address 127.0.97.3:33283)
0928 05:20:20.788756 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 05:20:20.788762 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:20:20.789089 (+ 327us) server_negotiation.cc:407] Connection header received
0928 05:20:20.789303 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:20:20.789309 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:20:20.789386 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:20:20.789467 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:20:20.790096 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:20.790880 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:20.792659 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:20.793420 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:20.793513 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:20.793613 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 05:20:20.793959 (+ 346us) server_negotiation.cc:299] Negotiation successful
0928 05:20:20.794079 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 05:20:20.794334 15251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:20.788195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33397 (local address 127.0.97.1:41813)
0928 05:20:20.788419 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 05:20:20.788426 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:20:20.789281 (+ 855us) server_negotiation.cc:407] Connection header received
0928 05:20:20.789517 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:20:20.789522 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:20:20.789597 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:20:20.789691 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:20:20.790285 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:20.790971 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:20.792680 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:20.793442 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:20.793567 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:20.793654 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 05:20:20.793974 (+ 320us) server_negotiation.cc:299] Negotiation successful
0928 05:20:20.794096 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 05:20:20.794466 15265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:20.788078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33283 (local address 127.0.97.2:37257)
0928 05:20:20.788969 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:20.788993 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 05:20:20.789160 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:20:20.789498 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:20:20.789502 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:20:20.789564 (+ 62us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:20:20.789963 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:20:20.789974 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:20:20.791003 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:20:20.791008 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:20:20.792512 (+ 1504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:20:20.792524 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:20:20.793641 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:20:20.793648 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:20:20.793820 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:20.793843 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 05:20:20.793917 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 05:20:20.794032 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":710,"mutex_wait_us":24,"thread_start_us":504,"threads_started":1}
I20260928 05:20:20.794492 15266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:20.788078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41813 (local address 127.0.97.2:33397)
0928 05:20:20.789194 (+ 1116us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:20.789207 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 05:20:20.789373 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:20:20.789713 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:20:20.789716 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:20:20.789738 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:20:20.790125 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:20:20.790135 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:20:20.791101 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:20:20.791105 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:20:20.792531 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:20:20.792542 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:20:20.793675 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:20:20.793681 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:20:20.793834 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:20.793856 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 05:20:20.793938 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 05:20:20.794057 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":999,"mutex_wait_us":82,"thread_start_us":777,"threads_started":1}
I20260928 05:20:20.795373 15195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72ba574bbee64b239d3c406e5fbde300" candidate_uuid: "3c747673187c42a894b270c6769ddab2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48927739d82c4f6a8274019312c386c6" is_pre_election: true
I20260928 05:20:20.795526 14932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72ba574bbee64b239d3c406e5fbde300" candidate_uuid: "3c747673187c42a894b270c6769ddab2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9fc7afd66a64bf997acf128e9f4ad28" is_pre_election: true
I20260928 05:20:20.795819 15195 raft_consensus.cc:2454] T 72ba574bbee64b239d3c406e5fbde300 P 48927739d82c4f6a8274019312c386c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c747673187c42a894b270c6769ddab2 in term 0.
I20260928 05:20:20.795986 14932 raft_consensus.cc:2454] T 72ba574bbee64b239d3c406e5fbde300 P d9fc7afd66a64bf997acf128e9f4ad28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c747673187c42a894b270c6769ddab2 in term 0.
I20260928 05:20:20.796416 14997 leader_election.cc:302] T 72ba574bbee64b239d3c406e5fbde300 P 3c747673187c42a894b270c6769ddab2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c747673187c42a894b270c6769ddab2, 48927739d82c4f6a8274019312c386c6; no voters:
I20260928 05:20:20.796756 15263 raft_consensus.cc:2781] T 72ba574bbee64b239d3c406e5fbde300 P 3c747673187c42a894b270c6769ddab2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 05:20:20.796912 15263 raft_consensus.cc:482] T 72ba574bbee64b239d3c406e5fbde300 P 3c747673187c42a894b270c6769ddab2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 05:20:20.797008 15263 raft_consensus.cc:3037] T 72ba574bbee64b239d3c406e5fbde300 P 3c747673187c42a894b270c6769ddab2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:20:20.799163 15263 raft_consensus.cc:504] T 72ba574bbee64b239d3c406e5fbde300 P 3c747673187c42a894b270c6769ddab2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9fc7afd66a64bf997acf128e9f4ad28" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41813 } } peers { permanent_uuid: "48927739d82c4f6a8274019312c386c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33283 } } peers { permanent_uuid: "3c747673187c42a894b270c6769ddab2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37923 } }
I20260928 05:20:20.799640 15263 leader_election.cc:288] T 72ba574bbee64b239d3c406e5fbde300 P 3c747673187c42a894b270c6769ddab2 [CANDIDATE]: Term 1 election: Requested vote from peers d9fc7afd66a64bf997acf128e9f4ad28 (127.0.97.1:41813), 48927739d82c4f6a8274019312c386c6 (127.0.97.3:33283)
I20260928 05:20:20.800076 14932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72ba574bbee64b239d3c406e5fbde300" candidate_uuid: "3c747673187c42a894b270c6769ddab2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9fc7afd66a64bf997acf128e9f4ad28"
I20260928 05:20:20.800237 14932 raft_consensus.cc:3037] T 72ba574bbee64b239d3c406e5fbde300 P d9fc7afd66a64bf997acf128e9f4ad28 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:20:20.801054 15195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72ba574bbee64b239d3c406e5fbde300" candidate_uuid: "3c747673187c42a894b270c6769ddab2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48927739d82c4f6a8274019312c386c6"
I20260928 05:20:20.801223 15195 raft_consensus.cc:3037] T 72ba574bbee64b239d3c406e5fbde300 P 48927739d82c4f6a8274019312c386c6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:20:20.802309 14932 raft_consensus.cc:2454] T 72ba574bbee64b239d3c406e5fbde300 P d9fc7afd66a64bf997acf128e9f4ad28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c747673187c42a894b270c6769ddab2 in term 1.
I20260928 05:20:20.802634 14999 leader_election.cc:302] T 72ba574bbee64b239d3c406e5fbde300 P 3c747673187c42a894b270c6769ddab2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c747673187c42a894b270c6769ddab2, d9fc7afd66a64bf997acf128e9f4ad28; no voters:
I20260928 05:20:20.802897 15263 raft_consensus.cc:2781] T 72ba574bbee64b239d3c406e5fbde300 P 3c747673187c42a894b270c6769ddab2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:20:20.803278 15263 raft_consensus.cc:686] T 72ba574bbee64b239d3c406e5fbde300 P 3c747673187c42a894b270c6769ddab2 [term 1 LEADER]: Becoming Leader. State: Replica: 3c747673187c42a894b270c6769ddab2, State: Running, Role: LEADER
I20260928 05:20:20.803751 15195 raft_consensus.cc:2454] T 72ba574bbee64b239d3c406e5fbde300 P 48927739d82c4f6a8274019312c386c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c747673187c42a894b270c6769ddab2 in term 1.
I20260928 05:20:20.803720 15263 consensus_queue.cc:237] T 72ba574bbee64b239d3c406e5fbde300 P 3c747673187c42a894b270c6769ddab2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9fc7afd66a64bf997acf128e9f4ad28" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41813 } } peers { permanent_uuid: "48927739d82c4f6a8274019312c386c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33283 } } peers { permanent_uuid: "3c747673187c42a894b270c6769ddab2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37923 } }
I20260928 05:20:20.807964 14792 catalog_manager.cc:5495] T 72ba574bbee64b239d3c406e5fbde300 P 3c747673187c42a894b270c6769ddab2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c747673187c42a894b270c6769ddab2 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "3c747673187c42a894b270c6769ddab2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9fc7afd66a64bf997acf128e9f4ad28" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48927739d82c4f6a8274019312c386c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c747673187c42a894b270c6769ddab2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37923 } health_report { overall_health: HEALTHY } } }
I20260928 05:20:20.879288 15255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:20.874355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54588 (local address 127.0.97.2:37923)
0928 05:20:20.874553 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 05:20:20.874559 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:20:20.874577 (+ 18us) server_negotiation.cc:407] Connection header received
0928 05:20:20.874722 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:20:20.874726 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:20:20.874811 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:20:20.874932 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:20:20.875287 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:20.875978 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:20.877323 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:20.877728 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:20.877831 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:20.878287 (+ 456us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:20:20.878758 (+ 471us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:20:20.878884 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 05:20:20.878976 (+ 92us) server_negotiation.cc:299] Negotiation successful
0928 05:20:20.879051 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 05:20:20.886840 15195 raft_consensus.cc:1261] T 72ba574bbee64b239d3c406e5fbde300 P 48927739d82c4f6a8274019312c386c6 [term 1 FOLLOWER]: Refusing update from remote peer 3c747673187c42a894b270c6769ddab2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:20:20.886857 14932 raft_consensus.cc:1261] T 72ba574bbee64b239d3c406e5fbde300 P d9fc7afd66a64bf997acf128e9f4ad28 [term 1 FOLLOWER]: Refusing update from remote peer 3c747673187c42a894b270c6769ddab2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:20:20.887705 15267 consensus_queue.cc:1035] T 72ba574bbee64b239d3c406e5fbde300 P 3c747673187c42a894b270c6769ddab2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9fc7afd66a64bf997acf128e9f4ad28" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:20:20.888010 15263 consensus_queue.cc:1035] T 72ba574bbee64b239d3c406e5fbde300 P 3c747673187c42a894b270c6769ddab2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48927739d82c4f6a8274019312c386c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:20:20.909566 14791 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:40970:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:20:20.909881 14791 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:40970:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:20:20.913460 14791 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3b36d06442754a95908b39ef6b46ad06: Sending DeleteTablet for 3 replicas of tablet 72ba574bbee64b239d3c406e5fbde300
I20260928 05:20:20.914459 15175 tablet_service.cc:1514] Processing DeleteTablet for tablet 72ba574bbee64b239d3c406e5fbde300 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:20:20 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41384
I20260928 05:20:20.914469 14912 tablet_service.cc:1514] Processing DeleteTablet for tablet 72ba574bbee64b239d3c406e5fbde300 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:20:20 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49770
I20260928 05:20:20.915035 15044 tablet_service.cc:1514] Processing DeleteTablet for tablet 72ba574bbee64b239d3c406e5fbde300 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:20:20 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54576
I20260928 05:20:20.915227 15280 tablet_replica.cc:330] T 72ba574bbee64b239d3c406e5fbde300 P d9fc7afd66a64bf997acf128e9f4ad28: stopping tablet replica
I20260928 05:20:20.915264 15279 tablet_replica.cc:330] T 72ba574bbee64b239d3c406e5fbde300 P 48927739d82c4f6a8274019312c386c6: stopping tablet replica
I20260928 05:20:20.915625 15280 raft_consensus.cc:2229] T 72ba574bbee64b239d3c406e5fbde300 P d9fc7afd66a64bf997acf128e9f4ad28 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:20:20.915721 15279 raft_consensus.cc:2229] T 72ba574bbee64b239d3c406e5fbde300 P 48927739d82c4f6a8274019312c386c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:20:20.916092 15280 raft_consensus.cc:2258] T 72ba574bbee64b239d3c406e5fbde300 P d9fc7afd66a64bf997acf128e9f4ad28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:20:20.916308 15279 raft_consensus.cc:2258] T 72ba574bbee64b239d3c406e5fbde300 P 48927739d82c4f6a8274019312c386c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:20:20.916476 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 14829
I20260928 05:20:20.918182 15280 ts_tablet_manager.cc:1905] T 72ba574bbee64b239d3c406e5fbde300 P d9fc7afd66a64bf997acf128e9f4ad28: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:20:20.922959 15279 ts_tablet_manager.cc:1905] T 72ba574bbee64b239d3c406e5fbde300 P 48927739d82c4f6a8274019312c386c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:20:20.925050 15280 ts_tablet_manager.cc:1918] T 72ba574bbee64b239d3c406e5fbde300 P d9fc7afd66a64bf997acf128e9f4ad28: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:20:20.925199 15280 log.cc:1196] T 72ba574bbee64b239d3c406e5fbde300 P d9fc7afd66a64bf997acf128e9f4ad28: Deleting WAL directory at /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-0/wal/wals/72ba574bbee64b239d3c406e5fbde300
I20260928 05:20:20.925720 15280 ts_tablet_manager.cc:1939] T 72ba574bbee64b239d3c406e5fbde300 P d9fc7afd66a64bf997acf128e9f4ad28: Deleting consensus metadata
I20260928 05:20:20.927392 14778 catalog_manager.cc:4841] TS d9fc7afd66a64bf997acf128e9f4ad28 (127.0.97.1:41813): tablet 72ba574bbee64b239d3c406e5fbde300 (table test-table [id=7507f5c88bf84b1198a15700cac9a7cc]) successfully deleted
I20260928 05:20:20.930244 15281 tablet_replica.cc:330] T 72ba574bbee64b239d3c406e5fbde300 P 3c747673187c42a894b270c6769ddab2: stopping tablet replica
I20260928 05:20:20.930581 15279 ts_tablet_manager.cc:1918] T 72ba574bbee64b239d3c406e5fbde300 P 48927739d82c4f6a8274019312c386c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:20:20.930624 15281 raft_consensus.cc:2229] T 72ba574bbee64b239d3c406e5fbde300 P 3c747673187c42a894b270c6769ddab2 [term 1 LEADER]: Raft consensus shutting down.
I20260928 05:20:20.930838 15279 log.cc:1196] T 72ba574bbee64b239d3c406e5fbde300 P 48927739d82c4f6a8274019312c386c6: Deleting WAL directory at /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-2/wal/wals/72ba574bbee64b239d3c406e5fbde300
I20260928 05:20:20.931236 15281 raft_consensus.cc:2258] T 72ba574bbee64b239d3c406e5fbde300 P 3c747673187c42a894b270c6769ddab2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:20:20.931339 15279 ts_tablet_manager.cc:1939] T 72ba574bbee64b239d3c406e5fbde300 P 48927739d82c4f6a8274019312c386c6: Deleting consensus metadata
I20260928 05:20:20.932646 15251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:20.917250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49776 (local address 127.0.97.1:41813)
0928 05:20:20.917405 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 05:20:20.917411 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:20:20.917506 (+ 95us) server_negotiation.cc:407] Connection header received
0928 05:20:20.917694 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:20:20.917699 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:20:20.917780 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:20:20.917880 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:20:20.919719 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:20.920842 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:20.922879 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:20.923528 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:20.923648 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:20.926774 (+ 3126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:20:20.926784 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:20:20.926799 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:20:20.926840 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:20:20.929683 (+ 2843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:20.930451 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:20.930459 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:20.930463 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:20.930586 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:20.930987 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:20.930992 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:20.930995 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:20.931362 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:20:20.931624 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0928 05:20:20.932134 (+ 510us) server_negotiation.cc:299] Negotiation successful
0928 05:20:20.932321 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 05:20:20.932792 14776 catalog_manager.cc:4841] TS 48927739d82c4f6a8274019312c386c6 (127.0.97.3:33283): tablet 72ba574bbee64b239d3c406e5fbde300 (table test-table [id=7507f5c88bf84b1198a15700cac9a7cc]) successfully deleted
I20260928 05:20:20.933037 15281 ts_tablet_manager.cc:1905] T 72ba574bbee64b239d3c406e5fbde300 P 3c747673187c42a894b270c6769ddab2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:20:20.933326 14972 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:20:20.939069 15281 ts_tablet_manager.cc:1918] T 72ba574bbee64b239d3c406e5fbde300 P 3c747673187c42a894b270c6769ddab2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:20:20.939266 15281 log.cc:1196] T 72ba574bbee64b239d3c406e5fbde300 P 3c747673187c42a894b270c6769ddab2: Deleting WAL directory at /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-1/wal/wals/72ba574bbee64b239d3c406e5fbde300
I20260928 05:20:20.939739 15281 ts_tablet_manager.cc:1939] T 72ba574bbee64b239d3c406e5fbde300 P 3c747673187c42a894b270c6769ddab2: Deleting consensus metadata
I20260928 05:20:20.941174 14779 catalog_manager.cc:4841] TS 3c747673187c42a894b270c6769ddab2 (127.0.97.2:37923): tablet 72ba574bbee64b239d3c406e5fbde300 (table test-table [id=7507f5c88bf84b1198a15700cac9a7cc]) successfully deleted
I20260928 05:20:21.033584 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 14829
I20260928 05:20:21.047406 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 14981
I20260928 05:20:21.060691 15255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:21.048208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54592 (local address 127.0.97.2:37923)
0928 05:20:21.048419 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 05:20:21.048428 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:20:21.048446 (+ 18us) server_negotiation.cc:407] Connection header received
0928 05:20:21.048504 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:20:21.048510 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:20:21.048591 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:20:21.048682 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:20:21.049961 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:21.050766 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:21.052025 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:21.052467 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:21.052574 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:21.055349 (+ 2775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:20:21.055359 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:20:21.055374 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:20:21.055406 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:20:21.058179 (+ 2773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:21.058769 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:21.058778 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:21.058782 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:21.058925 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:21.059279 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:21.059284 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:21.059287 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:21.059630 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:20:21.059912 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0928 05:20:21.060160 (+ 248us) server_negotiation.cc:299] Negotiation successful
0928 05:20:21.060364 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 05:20:21.061585 15104 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:20:21.168459 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 14981
I20260928 05:20:21.183737 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 15112
I20260928 05:20:21.199520 15254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:21.184361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41398 (local address 127.0.97.3:33283)
0928 05:20:21.184507 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 05:20:21.184511 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:20:21.184541 (+ 30us) server_negotiation.cc:407] Connection header received
0928 05:20:21.184706 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:20:21.184710 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:20:21.184772 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:20:21.184875 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:20:21.186079 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:21.186773 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:21.187923 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:21.188305 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:21.188397 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:21.194736 (+ 6339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:20:21.194749 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:20:21.194764 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:20:21.194809 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:20:21.197403 (+ 2594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:21.197939 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:21.197945 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:21.197948 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:21.198110 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:21.198378 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:21.198381 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:21.198383 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:21.198697 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:20:21.198896 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0928 05:20:21.199123 (+ 227us) server_negotiation.cc:299] Negotiation successful
0928 05:20:21.199271 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 05:20:21.200124 15235 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:20:21.285573 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 15112
I20260928 05:20:21.298527 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 14759
I20260928 05:20:21.299566 14822 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:20:21.409638 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 14759
I20260928 05:20:21.426611 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38873
--webserver_interface=127.0.97.62
--webserver_port=40133
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38873
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:20:21.736258 15286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:20:21.736549 15286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:20:21.736651 15286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:20:21.743837 15286 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:20:21.743939 15286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:20:21.743997 15286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:20:21.744059 15286 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:20:21.744108 15286 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:20:21.753644 15286 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38873
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=40133
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:20:21.754796 15286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:20:21.756011 15286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:20:21.766987 15291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:20:21.766921 15292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:20:21.767019 15294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:20:21.767831 15286 server_base.cc:1034] running on GCE node
I20260928 05:20:21.768469 15286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:20:21.769719 15286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:20:21.770903 15286 hybrid_clock.cc:648] HybridClock initialized: now 1790572821770856 us; error 32 us; skew 500 ppm
I20260928 05:20:21.771190 15286 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:20:21 dist-test-slave-whsn krb5kdc[14209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572821, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:20:21.773940 15286 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 05:20:21.775265 15286 webserver.cc:460] Webserver started at http://127.0.97.62:40133/ using document root <none> and password file <none>
I20260928 05:20:21.775861 15286 fs_manager.cc:359] Metadata directory not provided
I20260928 05:20:21.776005 15286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:20:21.780934 15286 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 05:20:21.783430 15301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:20:21.784578 15286 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 05:20:21.784726 15286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/master-0/wal
uuid: "3b36d06442754a95908b39ef6b46ad06"
format_stamp: "Formatted at 2026-09-28 05:20:16 on dist-test-slave-whsn"
server_key: "e98dabea40fe88d3c480d40e6b9ff037"
server_key_iv: "77733407db60baee2278fc1927cf2899"
server_key_version: "encryptionkey@0"
I20260928 05:20:21.785647 15286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:20:21.804648 15286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:20:21.808065 15286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:20:21.808290 15286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:20:21.824362 15286 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38873
I20260928 05:20:21.824393 15352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38873 every 8 connection(s)
I20260928 05:20:21.825697 15286 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:20:21.831859 15353 sys_catalog.cc:263] Verifying existing consensus state
I20260928 05:20:21.833637 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 15286
I20260928 05:20:21.834556 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:41813
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:38873
--webserver_port=42701
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 05:20:21.835594 15353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b36d06442754a95908b39ef6b46ad06: Bootstrap starting.
I20260928 05:20:21.856276 15353 log.cc:824] T 00000000000000000000000000000000 P 3b36d06442754a95908b39ef6b46ad06: Log is configured to *not* fsync() on all Append() calls
I20260928 05:20:21.871728 15353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b36d06442754a95908b39ef6b46ad06: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 05:20:21.872187 15353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b36d06442754a95908b39ef6b46ad06: Bootstrap complete.
I20260928 05:20:21.880620 15353 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b36d06442754a95908b39ef6b46ad06 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b36d06442754a95908b39ef6b46ad06" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38873 } }
I20260928 05:20:21.881479 15353 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b36d06442754a95908b39ef6b46ad06 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b36d06442754a95908b39ef6b46ad06, State: Initialized, Role: FOLLOWER
I20260928 05:20:21.882066 15353 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b36d06442754a95908b39ef6b46ad06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b36d06442754a95908b39ef6b46ad06" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38873 } }
I20260928 05:20:21.882283 15353 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b36d06442754a95908b39ef6b46ad06 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:20:21.882392 15353 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b36d06442754a95908b39ef6b46ad06 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:20:21.882543 15353 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b36d06442754a95908b39ef6b46ad06 [term 2 FOLLOWER]: Advancing to term 3
I20260928 05:20:21.886201 15353 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b36d06442754a95908b39ef6b46ad06 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b36d06442754a95908b39ef6b46ad06" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38873 } }
I20260928 05:20:21.886677 15353 leader_election.cc:302] T 00000000000000000000000000000000 P 3b36d06442754a95908b39ef6b46ad06 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b36d06442754a95908b39ef6b46ad06; no voters:
I20260928 05:20:21.887253 15353 leader_election.cc:288] T 00000000000000000000000000000000 P 3b36d06442754a95908b39ef6b46ad06 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 05:20:21.887382 15357 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b36d06442754a95908b39ef6b46ad06 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 05:20:21.888410 15357 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b36d06442754a95908b39ef6b46ad06 [term 3 LEADER]: Becoming Leader. State: Replica: 3b36d06442754a95908b39ef6b46ad06, State: Running, Role: LEADER
I20260928 05:20:21.888983 15357 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b36d06442754a95908b39ef6b46ad06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b36d06442754a95908b39ef6b46ad06" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38873 } }
I20260928 05:20:21.889288 15353 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b36d06442754a95908b39ef6b46ad06 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:20:21.891539 15359 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b36d06442754a95908b39ef6b46ad06 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b36d06442754a95908b39ef6b46ad06. Latest consensus state: current_term: 3 leader_uuid: "3b36d06442754a95908b39ef6b46ad06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b36d06442754a95908b39ef6b46ad06" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38873 } } }
I20260928 05:20:21.891551 15358 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b36d06442754a95908b39ef6b46ad06 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "3b36d06442754a95908b39ef6b46ad06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b36d06442754a95908b39ef6b46ad06" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38873 } } }
I20260928 05:20:21.892016 15358 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b36d06442754a95908b39ef6b46ad06 [sys.catalog]: This master's current role is: LEADER
I20260928 05:20:21.891924 15359 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b36d06442754a95908b39ef6b46ad06 [sys.catalog]: This master's current role is: LEADER
I20260928 05:20:21.896200 15362 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:20:21.904687 15362 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:20:21.906164 15362 catalog_manager.cc:1259] Loaded cluster ID: 300d65fd8cab49eda278356e8e700e03
I20260928 05:20:21.906277 15362 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260928 05:20:21.908278 15362 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f50fc6dd9ad google::LogMessage::Fail() at ??:0
@ 0x7f50fc6e088a google::LogMessage::SendToLog() at ??:0
@ 0x7f50fc6dd4de google::LogMessage::Flush() at ??:0
@ 0x7f50fc6dea19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f5112110512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f511210d976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f5112195610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f5112195439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f50ff329345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f50ff32c91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
I20260928 05:20:21.944881 15377 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7f50ff32c799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f50ff307eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f51025e16db start_thread at ??:0
@ 0x7f50fb57761f clone at ??:0
W20260928 05:20:22.197646 15355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:20:22.197901 15355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:20:22.198012 15355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:20:22.205055 15355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:20:22.205152 15355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:20:22.205273 15355 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 05:20:22.214591 15355 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:41813
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=42701
--tserver_master_addrs=127.0.97.62:38873
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:20:22.215691 15355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:20:22.216936 15355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:20:22.229061 15394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:20:22.229142 15397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:20:22.229097 15395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:20:22.230163 15355 server_base.cc:1034] running on GCE node
I20260928 05:20:22.230746 15355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:20:22.231590 15355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:20:22.232800 15355 hybrid_clock.cc:648] HybridClock initialized: now 1790572822232757 us; error 32 us; skew 500 ppm
I20260928 05:20:22.233127 15355 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:20:22 dist-test-slave-whsn krb5kdc[14209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572822, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:20:22.236035 15355 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260928 05:20:22.237378 15355 webserver.cc:460] Webserver started at http://127.0.97.1:42701/ using document root <none> and password file <none>
I20260928 05:20:22.237995 15355 fs_manager.cc:359] Metadata directory not provided
I20260928 05:20:22.238106 15355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:20:22.243033 15355 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 05:20:22.245548 15404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:20:22.246832 15355 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:20:22.246984 15355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-0/wal
uuid: "d9fc7afd66a64bf997acf128e9f4ad28"
format_stamp: "Formatted at 2026-09-28 05:20:16 on dist-test-slave-whsn"
server_key: "abfff00c90c1ec802b746e533217ed74"
server_key_iv: "6bdbfa0c04b72d97b3fe600767c65655"
server_key_version: "encryptionkey@0"
I20260928 05:20:22.247895 15355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:20:22.270170 15355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:20:22.273663 15355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:20:22.273900 15355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:20:22.274775 15355 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:20:22.275992 15355 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:20:22.276094 15355 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:20:22.276192 15355 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:20:22.276261 15355 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:20:22.305675 15355 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41813
I20260928 05:20:22.305750 15516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41813 every 8 connection(s)
I20260928 05:20:22.307050 15355 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-0/data/info.pb
I20260928 05:20:22.308027 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 15355
I20260928 05:20:22.308745 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:37923
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:38873
--webserver_port=37261
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 05:20:22.312271 15519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:22.310149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38873 (local address 127.0.97.1:39057)
0928 05:20:22.310936 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:22.311192 (+ 256us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":285,"threads_started":1}
W20260928 05:20:22.313004 15517 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:38873 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:38873: Client connection negotiation failed: client connection to 127.0.97.62:38873: connect: Connection refused (error 111)
I20260928 05:20:22.313776 15519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:22.313454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38873 (local address 127.0.97.1:54067)
0928 05:20:22.313589 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:22.313659 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260928 05:20:22.314548 15519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:22.314231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38873 (local address 127.0.97.1:34081)
0928 05:20:22.314351 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:22.314429 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260928 05:20:22.314749 15517 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 05:20:22.610265 15520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:20:22.610545 15520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:20:22.610680 15520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:20:22.617568 15520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:20:22.617666 15520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:20:22.617813 15520 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 05:20:22.626782 15520 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:37923
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=37261
--tserver_master_addrs=127.0.97.62:38873
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:20:22.627876 15520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:20:22.629077 15520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:20:22.640586 15525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:20:22.640604 15526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:20:22.640712 15528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:20:22.641330 15520 server_base.cc:1034] running on GCE node
I20260928 05:20:22.641901 15520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:20:22.642736 15520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:20:22.643944 15520 hybrid_clock.cc:648] HybridClock initialized: now 1790572822643904 us; error 32 us; skew 500 ppm
I20260928 05:20:22.644220 15520 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:20:22 dist-test-slave-whsn krb5kdc[14209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572822, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:20:22.647081 15520 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260928 05:20:22.648313 15520 webserver.cc:460] Webserver started at http://127.0.97.2:37261/ using document root <none> and password file <none>
I20260928 05:20:22.648958 15520 fs_manager.cc:359] Metadata directory not provided
I20260928 05:20:22.649068 15520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:20:22.653998 15520 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260928 05:20:22.656348 15535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:20:22.657529 15520 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 05:20:22.657685 15520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-1/wal
uuid: "3c747673187c42a894b270c6769ddab2"
format_stamp: "Formatted at 2026-09-28 05:20:17 on dist-test-slave-whsn"
server_key: "b3f72ace898d07ce94f3ff18a606b377"
server_key_iv: "dc2d72eee7b3063c3403165fe5fbdcf1"
server_key_version: "encryptionkey@0"
I20260928 05:20:22.658572 15520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:20:22.677093 15520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:20:22.680485 15520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:20:22.680722 15520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:20:22.681648 15520 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:20:22.682860 15520 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:20:22.682963 15520 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:20:22.683069 15520 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:20:22.683137 15520 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:20:22.711490 15520 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37923
I20260928 05:20:22.711551 15647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37923 every 8 connection(s)
I20260928 05:20:22.712874 15520 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-1/data/info.pb
I20260928 05:20:22.717284 15650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:22.715506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38873 (local address 127.0.97.2:58709)
0928 05:20:22.716199 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:22.716381 (+ 182us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":262,"threads_started":1}
W20260928 05:20:22.717906 15648 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:38873 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:38873: Client connection negotiation failed: client connection to 127.0.97.62:38873: connect: Connection refused (error 111)
I20260928 05:20:22.718242 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 15520
I20260928 05:20:22.718628 15650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:22.718348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38873 (local address 127.0.97.2:33253)
0928 05:20:22.718468 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:22.718532 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260928 05:20:22.718827 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:33283
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:38873
--webserver_port=41963
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 05:20:22.719383 15650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:22.719090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38873 (local address 127.0.97.2:43275)
0928 05:20:22.719218 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:22.719286 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260928 05:20:22.719584 15648 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 05:20:23.020769 15651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:20:23.021068 15651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:20:23.021210 15651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:20:23.028424 15651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:20:23.028523 15651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:20:23.028676 15651 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 05:20:23.038159 15651 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:33283
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=41963
--tserver_master_addrs=127.0.97.62:38873
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:20:23.039309 15651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:20:23.040512 15651 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:20:23.052655 15656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:20:23.052603 15659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:20:23.052676 15657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:20:23.053694 15651 server_base.cc:1034] running on GCE node
I20260928 05:20:23.054252 15651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:20:23.055066 15651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:20:23.056301 15651 hybrid_clock.cc:648] HybridClock initialized: now 1790572823056253 us; error 35 us; skew 500 ppm
I20260928 05:20:23.056627 15651 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:20:23 dist-test-slave-whsn krb5kdc[14209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572823, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:20:23.059945 15651 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260928 05:20:23.061470 15651 webserver.cc:460] Webserver started at http://127.0.97.3:41963/ using document root <none> and password file <none>
I20260928 05:20:23.062131 15651 fs_manager.cc:359] Metadata directory not provided
I20260928 05:20:23.062238 15651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:20:23.067488 15651 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260928 05:20:23.070062 15666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:20:23.071292 15651 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:20:23.071425 15651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-2/wal
uuid: "48927739d82c4f6a8274019312c386c6"
format_stamp: "Formatted at 2026-09-28 05:20:17 on dist-test-slave-whsn"
server_key: "a9c2679720605cdf6dedaff6d97079d3"
server_key_iv: "87e381d6de55e57ee2203c2d2cd41123"
server_key_version: "encryptionkey@0"
I20260928 05:20:23.072350 15651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:20:23.085855 15651 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:20:23.089957 15651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:20:23.090205 15651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:20:23.091127 15651 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:20:23.092428 15651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:20:23.092518 15651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:20:23.092602 15651 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:20:23.092659 15651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:20:23.123901 15651 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33283
I20260928 05:20:23.123950 15778 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33283 every 8 connection(s)
I20260928 05:20:23.125408 15651 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572622576160-388-0/minicluster-data/ts-2/data/info.pb
I20260928 05:20:23.127419 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 15651
I20260928 05:20:23.130229 15782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:23.128223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38873 (local address 127.0.97.3:44459)
0928 05:20:23.129062 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:23.129316 (+ 254us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":343,"threads_started":1}
W20260928 05:20:23.130862 15779 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:38873 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:38873: Client connection negotiation failed: client connection to 127.0.97.62:38873: connect: Connection refused (error 111)
I20260928 05:20:23.131624 15782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:23.131290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38873 (local address 127.0.97.3:43427)
0928 05:20:23.131435 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:23.131521 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 05:20:23.132368 15782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:23.132067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38873 (local address 127.0.97.3:45393)
0928 05:20:23.132196 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:23.132268 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260928 05:20:23.132573 15779 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 05:20:23.315877 15783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:23.315183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38873 (local address 127.0.97.1:47309)
0928 05:20:23.315591 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:23.315670 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":199,"threads_started":1}
I20260928 05:20:23.720737 15784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:23.720033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38873 (local address 127.0.97.2:48493)
0928 05:20:23.720435 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:23.720513 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":191,"threads_started":1}
I20260928 05:20:24.133777 15785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:24.133072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38873 (local address 127.0.97.3:52371)
0928 05:20:24.133481 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:24.133560 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":204,"threads_started":1}
I20260928 05:20:24.317248 15786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:24.316571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38873 (local address 127.0.97.1:46335)
0928 05:20:24.317026 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:24.317096 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":227,"threads_started":1}
I20260928 05:20:24.722148 15787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:24.721522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38873 (local address 127.0.97.2:34151)
0928 05:20:24.721917 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:24.721997 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":195,"threads_started":1}
I20260928 05:20:25.135162 15788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:25.134525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38873 (local address 127.0.97.3:55759)
0928 05:20:25.134930 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:25.135009 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":204,"threads_started":1}
I20260928 05:20:25.318578 15789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:25.317952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38873 (local address 127.0.97.1:47583)
0928 05:20:25.318356 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:25.318425 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":215,"threads_started":1}
I20260928 05:20:25.723559 15790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:25.722916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38873 (local address 127.0.97.2:38271)
0928 05:20:25.723317 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:25.723394 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":213,"threads_started":1}
I20260928 05:20:26.136474 15791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:26.135864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38873 (local address 127.0.97.3:51935)
0928 05:20:26.136248 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:26.136320 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":202,"threads_started":1}
I20260928 05:20:26.319969 15792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:26.319299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38873 (local address 127.0.97.1:44627)
0928 05:20:26.319708 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:26.319791 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":199,"threads_started":1}
I20260928 05:20:26.724977 15793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:26.724281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38873 (local address 127.0.97.2:56557)
0928 05:20:26.724678 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:26.724755 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":195,"threads_started":1}
I20260928 05:20:27.137861 15794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:27.137235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38873 (local address 127.0.97.3:33387)
0928 05:20:27.137612 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:27.137684 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":187,"threads_started":1}
I20260928 05:20:27.321400 15795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:27.320740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38873 (local address 127.0.97.1:54595)
0928 05:20:27.321161 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:27.321243 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":214,"threads_started":1}
I20260928 05:20:27.726418 15796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:27.725755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38873 (local address 127.0.97.2:35999)
0928 05:20:27.726172 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:27.726245 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":212,"threads_started":1}
I20260928 05:20:28.139277 15797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:28.138592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38873 (local address 127.0.97.3:60829)
0928 05:20:28.139026 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:28.139116 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":214,"threads_started":1}
I20260928 05:20:28.322762 15798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:28.322110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38873 (local address 127.0.97.1:57701)
0928 05:20:28.322498 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:28.322603 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":201,"threads_started":1}
I20260928 05:20:28.727825 15799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:28.727140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38873 (local address 127.0.97.2:43975)
0928 05:20:28.727584 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:28.727664 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":250,"threads_started":1}
I20260928 05:20:29.140677 15800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:29.140007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38873 (local address 127.0.97.3:39311)
0928 05:20:29.140435 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:29.140516 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":226,"threads_started":1}
I20260928 05:20:29.324319 15801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:29.323521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38873 (local address 127.0.97.1:58399)
0928 05:20:29.324039 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:29.324110 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":291,"threads_started":1}
I20260928 05:20:29.729452 15802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:29.728667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38873 (local address 127.0.97.2:54185)
0928 05:20:29.729167 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:29.729243 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":232,"threads_started":1}
I20260928 05:20:30.142294 15803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:30.141513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38873 (local address 127.0.97.3:50599)
0928 05:20:30.142030 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:30.142101 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":229,"threads_started":1}
I20260928 05:20:30.325773 15804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:30.325126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38873 (local address 127.0.97.1:56205)
0928 05:20:30.325542 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:30.325614 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":199,"threads_started":1}
I20260928 05:20:30.730878 15805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:30.730260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38873 (local address 127.0.97.2:40147)
0928 05:20:30.730654 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:30.730726 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":204,"threads_started":1}
I20260928 05:20:31.143682 15806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:31.143029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38873 (local address 127.0.97.3:50377)
0928 05:20:31.143447 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:31.143522 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":220,"threads_started":1}
I20260928 05:20:31.327332 15807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:31.326558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38873 (local address 127.0.97.1:37945)
0928 05:20:31.327074 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:31.327158 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":279,"threads_started":1}
I20260928 05:20:31.732362 15808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:31.731639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38873 (local address 127.0.97.2:58603)
0928 05:20:31.732108 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:31.732192 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":230,"threads_started":1}
I20260928 05:20:32.145270 15809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:32.144480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38873 (local address 127.0.97.3:44641)
0928 05:20:32.145025 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:32.145099 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":254,"threads_started":1}
I20260928 05:20:32.328794 15810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:32.328124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38873 (local address 127.0.97.1:56879)
0928 05:20:32.328560 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:32.328633 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":232,"threads_started":1}
I20260928 05:20:32.734047 15811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:32.733374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38873 (local address 127.0.97.2:59273)
0928 05:20:32.733800 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:32.733891 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":227,"threads_started":1}
I20260928 05:20:33.146713 15812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:33.146054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38873 (local address 127.0.97.3:33285)
0928 05:20:33.146479 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:33.146555 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":225,"threads_started":1}
I20260928 05:20:33.330410 15813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:33.329642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38873 (local address 127.0.97.1:54461)
0928 05:20:33.330114 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:33.330215 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":227,"threads_started":1}
I20260928 05:20:33.735584 15814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:33.734881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38873 (local address 127.0.97.2:50901)
0928 05:20:33.735328 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:33.735411 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":254,"threads_started":1}
I20260928 05:20:34.148144 15815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:34.147501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38873 (local address 127.0.97.3:45721)
0928 05:20:34.147904 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:34.147987 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":203,"threads_started":1}
I20260928 05:20:34.331881 15816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:34.331241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38873 (local address 127.0.97.1:46721)
0928 05:20:34.331648 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:34.331729 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":222,"threads_started":1}
I20260928 05:20:34.737051 15817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:34.736411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38873 (local address 127.0.97.2:43443)
0928 05:20:34.736790 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:34.736892 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":183,"threads_started":1}
I20260928 05:20:35.149590 15818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:35.148989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38873 (local address 127.0.97.3:43923)
0928 05:20:35.149371 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:35.149444 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":201,"threads_started":1}
I20260928 05:20:35.333387 15819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:35.332658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38873 (local address 127.0.97.1:44255)
0928 05:20:35.333121 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:35.333203 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":250,"threads_started":1}
I20260928 05:20:35.738693 15820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:35.737965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38873 (local address 127.0.97.2:35379)
0928 05:20:35.738420 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:35.738507 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":212,"threads_started":1}
I20260928 05:20:36.151306 15821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:36.150522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38873 (local address 127.0.97.3:59239)
0928 05:20:36.151023 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:36.151111 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":276,"threads_started":1}
I20260928 05:20:36.335083 15822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:36.334345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38873 (local address 127.0.97.1:52527)
0928 05:20:36.334813 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:36.334900 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":236,"threads_started":1}
I20260928 05:20:36.740111 15823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:36.739435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38873 (local address 127.0.97.2:33615)
0928 05:20:36.739861 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:36.739935 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":215,"threads_started":1}
I20260928 05:20:37.152695 15824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:37.152031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38873 (local address 127.0.97.3:47957)
0928 05:20:37.152448 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:37.152521 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":241,"threads_started":1}
I20260928 05:20:37.336735 15825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:37.336033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38873 (local address 127.0.97.1:42453)
0928 05:20:37.336438 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:37.336532 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":217,"threads_started":1}
I20260928 05:20:37.741892 15826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:37.741156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38873 (local address 127.0.97.2:53165)
0928 05:20:37.741625 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:37.741705 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":267,"threads_started":1}
I20260928 05:20:38.128516 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 15355
I20260928 05:20:38.148097 15827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:38.129518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33918 (local address 127.0.97.1:41813)
0928 05:20:38.130059 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0928 05:20:38.130077 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 05:20:38.130132 (+ 55us) server_negotiation.cc:407] Connection header received
0928 05:20:38.131769 (+ 1637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:20:38.131778 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:20:38.132199 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:20:38.132474 (+ 275us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:20:38.133839 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:38.135080 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:38.136228 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:38.136995 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:38.137123 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:38.142778 (+ 5655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:20:38.142785 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:20:38.142800 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:20:38.142839 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:20:38.145395 (+ 2556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:38.146008 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:38.146014 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:38.146020 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:38.146168 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:38.146463 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:38.146466 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:38.146468 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:38.146936 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:20:38.147156 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0928 05:20:38.147448 (+ 292us) server_negotiation.cc:299] Negotiation successful
0928 05:20:38.147828 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":248,"threads_started":1}
I20260928 05:20:38.149461 15512 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:20:38.154438 15829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:38.153679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38873 (local address 127.0.97.3:41225)
0928 05:20:38.154119 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:38.154219 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":196,"threads_started":1}
I20260928 05:20:38.231616 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 15355
I20260928 05:20:38.245345 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 15520
I20260928 05:20:38.261660 15830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:38.246137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56916 (local address 127.0.97.2:37923)
0928 05:20:38.246629 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0928 05:20:38.246642 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 05:20:38.246680 (+ 38us) server_negotiation.cc:407] Connection header received
0928 05:20:38.248096 (+ 1416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:20:38.248105 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:20:38.248524 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:20:38.248805 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:20:38.250074 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:38.251352 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:38.252606 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:38.253378 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:38.253504 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:38.255995 (+ 2491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:20:38.256002 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:20:38.256018 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:20:38.256057 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:20:38.258862 (+ 2805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:38.259488 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:38.259494 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:38.259500 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:38.259656 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:38.259920 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:38.259923 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:38.259925 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:38.260393 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:20:38.260612 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0928 05:20:38.261041 (+ 429us) server_negotiation.cc:299] Negotiation successful
0928 05:20:38.261359 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":213,"threads_started":1}
I20260928 05:20:38.263087 15643 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:20:38.344040 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 15520
I20260928 05:20:38.357741 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 15651
I20260928 05:20:38.373816 15832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:38.358611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48742 (local address 127.0.97.3:33283)
0928 05:20:38.359174 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0928 05:20:38.359188 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 05:20:38.359228 (+ 40us) server_negotiation.cc:407] Connection header received
0928 05:20:38.360832 (+ 1604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:20:38.360842 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:20:38.361309 (+ 467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:20:38.361590 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:20:38.362812 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:38.364049 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:38.365305 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:38.365952 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:38.366087 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:38.368372 (+ 2285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:20:38.368379 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:20:38.368392 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:20:38.368429 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:20:38.370996 (+ 2567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:38.371652 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:38.371658 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:38.371664 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:38.371822 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:38.372131 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:38.372134 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:38.372136 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:38.372627 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:20:38.372904 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0928 05:20:38.373221 (+ 317us) server_negotiation.cc:299] Negotiation successful
0928 05:20:38.373531 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":246,"threads_started":1}
I20260928 05:20:38.375206 15774 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:20:38.458003 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 15651
2026-09-28T05:20:38Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (23850 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:20:38 dist-test-slave-whsn krb5kdc[15836](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:20:38 dist-test-slave-whsn krb5kdc[15836](info): set up 2 sockets
Sep 28 05:20:38 dist-test-slave-whsn krb5kdc[15836](info): commencing operation
krb5kdc: starting...
W20260928 05:20:39.461112 388 mini_kdc.cc:121] Time spent starting KDC: real 0.968s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/krb5kdc/test-user.keytab.
Sep 28 05:20:39 dist-test-slave-whsn krb5kdc[15836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572839, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:20:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:20:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 05:20:39.661755 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:32969
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:32969
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:20:39.970515 15849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:20:39.970790 15849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:20:39.970922 15849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:20:39.977785 15849 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 05:20:39.977890 15849 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 05:20:39.977941 15849 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 05:20:39.978010 15849 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:20:39.978086 15849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:20:39.978176 15849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:20:39.978236 15849 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:20:39.978310 15849 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:20:39.987897 15849 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:32969
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:32969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:20:39.989144 15849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:20:39.990424 15849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:20:40.001294 15854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:20:40.001394 15857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:20:40.001406 15855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:20:40.002272 15849 server_base.cc:1034] running on GCE node
I20260928 05:20:40.002990 15849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:20:40.004300 15849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:20:40.005533 15849 hybrid_clock.cc:648] HybridClock initialized: now 1790572840005475 us; error 38 us; skew 500 ppm
I20260928 05:20:40.005843 15849 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:20:40 dist-test-slave-whsn krb5kdc[15836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572840, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:20:40.008731 15849 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 05:20:40.010105 15849 webserver.cc:460] Webserver started at http://127.0.97.62:36021/ using document root <none> and password file <none>
I20260928 05:20:40.010732 15849 fs_manager.cc:359] Metadata directory not provided
I20260928 05:20:40.010820 15849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:20:40.011104 15849 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:20:40.013955 15849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/master-0/data/instance:
uuid: "fa08fd07ee1e42f8956f8a8464523f55"
format_stamp: "Formatted at 2026-09-28 05:20:40 on dist-test-slave-whsn"
server_key: "5283b56f3149cd0cafcc4d4b183d8b08"
server_key_iv: "0e3b1c8c08bb8235dec625a117da870f"
server_key_version: "encryptionkey@0"
I20260928 05:20:40.014653 15849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/master-0/wal/instance:
uuid: "fa08fd07ee1e42f8956f8a8464523f55"
format_stamp: "Formatted at 2026-09-28 05:20:40 on dist-test-slave-whsn"
server_key: "5283b56f3149cd0cafcc4d4b183d8b08"
server_key_iv: "0e3b1c8c08bb8235dec625a117da870f"
server_key_version: "encryptionkey@0"
I20260928 05:20:40.019325 15849 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 05:20:40.022447 15864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:20:40.023597 15849 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 05:20:40.023739 15849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/master-0/wal
uuid: "fa08fd07ee1e42f8956f8a8464523f55"
format_stamp: "Formatted at 2026-09-28 05:20:40 on dist-test-slave-whsn"
server_key: "5283b56f3149cd0cafcc4d4b183d8b08"
server_key_iv: "0e3b1c8c08bb8235dec625a117da870f"
server_key_version: "encryptionkey@0"
I20260928 05:20:40.023913 15849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:20:40.039710 15849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:20:40.043463 15849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:20:40.043676 15849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:20:40.060333 15849 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:32969
I20260928 05:20:40.060379 15915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:32969 every 8 connection(s)
I20260928 05:20:40.061609 15849 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:20:40.065135 15916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:20:40.070091 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 15849
I20260928 05:20:40.070307 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/master-0/wal/instance
I20260928 05:20:40.071084 388 external_mini_cluster.cc:1428] Setting key 78a99f451b63e72685e667613217a122
I20260928 05:20:40.076032 15916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa08fd07ee1e42f8956f8a8464523f55: Bootstrap starting.
I20260928 05:20:40.079468 15916 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fa08fd07ee1e42f8956f8a8464523f55: Neither blocks nor log segments found. Creating new log.
I20260928 05:20:40.080402 15916 log.cc:824] T 00000000000000000000000000000000 P fa08fd07ee1e42f8956f8a8464523f55: Log is configured to *not* fsync() on all Append() calls
I20260928 05:20:40.083029 15916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa08fd07ee1e42f8956f8a8464523f55: No bootstrap required, opened a new log
Sep 28 05:20:40 dist-test-slave-whsn krb5kdc[15836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572839, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:20:40.090267 15916 raft_consensus.cc:348] T 00000000000000000000000000000000 P fa08fd07ee1e42f8956f8a8464523f55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa08fd07ee1e42f8956f8a8464523f55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32969 } }
I20260928 05:20:40.090608 15916 raft_consensus.cc:374] T 00000000000000000000000000000000 P fa08fd07ee1e42f8956f8a8464523f55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:20:40.090706 15916 raft_consensus.cc:729] T 00000000000000000000000000000000 P fa08fd07ee1e42f8956f8a8464523f55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa08fd07ee1e42f8956f8a8464523f55, State: Initialized, Role: FOLLOWER
I20260928 05:20:40.091405 15916 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa08fd07ee1e42f8956f8a8464523f55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa08fd07ee1e42f8956f8a8464523f55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32969 } }
I20260928 05:20:40.091681 15916 raft_consensus.cc:388] T 00000000000000000000000000000000 P fa08fd07ee1e42f8956f8a8464523f55 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:20:40.091881 15916 raft_consensus.cc:482] T 00000000000000000000000000000000 P fa08fd07ee1e42f8956f8a8464523f55 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:20:40.092069 15916 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa08fd07ee1e42f8956f8a8464523f55 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:20:40.092818 15919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:40.073305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38822 (local address 127.0.97.62:32969)
0928 05:20:40.074111 (+ 806us) server_negotiation.cc:206] Beginning negotiation
0928 05:20:40.074125 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 05:20:40.074167 (+ 42us) server_negotiation.cc:407] Connection header received
0928 05:20:40.075779 (+ 1612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:20:40.075801 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:20:40.076191 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:20:40.076507 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:20:40.078231 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:40.079540 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:40.080889 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:40.081611 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:40.081784 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:40.085619 (+ 3835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:20:40.085626 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:20:40.085640 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:20:40.085678 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:20:40.088615 (+ 2937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:40.089313 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:40.089319 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:40.089325 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:40.089481 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:40.089829 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:40.089832 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:40.089834 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:40.090326 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:20:40.090607 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0928 05:20:40.091051 (+ 444us) server_negotiation.cc:299] Negotiation successful
0928 05:20:40.091481 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":379,"threads_started":1}
I20260928 05:20:40.094537 15916 raft_consensus.cc:504] T 00000000000000000000000000000000 P fa08fd07ee1e42f8956f8a8464523f55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa08fd07ee1e42f8956f8a8464523f55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32969 } }
I20260928 05:20:40.094991 15916 leader_election.cc:302] T 00000000000000000000000000000000 P fa08fd07ee1e42f8956f8a8464523f55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fa08fd07ee1e42f8956f8a8464523f55; no voters:
I20260928 05:20:40.095592 15916 leader_election.cc:288] T 00000000000000000000000000000000 P fa08fd07ee1e42f8956f8a8464523f55 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:20:40.095706 15921 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fa08fd07ee1e42f8956f8a8464523f55 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:20:40.096321 15921 raft_consensus.cc:686] T 00000000000000000000000000000000 P fa08fd07ee1e42f8956f8a8464523f55 [term 1 LEADER]: Becoming Leader. State: Replica: fa08fd07ee1e42f8956f8a8464523f55, State: Running, Role: LEADER
I20260928 05:20:40.096817 15921 consensus_queue.cc:237] T 00000000000000000000000000000000 P fa08fd07ee1e42f8956f8a8464523f55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa08fd07ee1e42f8956f8a8464523f55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32969 } }
I20260928 05:20:40.097503 15916 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa08fd07ee1e42f8956f8a8464523f55 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:20:40.099455 15922 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa08fd07ee1e42f8956f8a8464523f55 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fa08fd07ee1e42f8956f8a8464523f55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa08fd07ee1e42f8956f8a8464523f55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32969 } } }
I20260928 05:20:40.099661 15923 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa08fd07ee1e42f8956f8a8464523f55 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa08fd07ee1e42f8956f8a8464523f55. Latest consensus state: current_term: 1 leader_uuid: "fa08fd07ee1e42f8956f8a8464523f55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa08fd07ee1e42f8956f8a8464523f55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32969 } } }
I20260928 05:20:40.100015 15922 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa08fd07ee1e42f8956f8a8464523f55 [sys.catalog]: This master's current role is: LEADER
I20260928 05:20:40.100047 15923 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa08fd07ee1e42f8956f8a8464523f55 [sys.catalog]: This master's current role is: LEADER
I20260928 05:20:40.101774 15928 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:20:40.106964 15928 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:20:40.115407 15928 catalog_manager.cc:1347] Generated new cluster ID: 267ca691bdf34212ba7e1a115574519b
I20260928 05:20:40.115514 15928 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:20:40.142624 15928 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:20:40.144470 15928 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:20:40.154213 15928 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fa08fd07ee1e42f8956f8a8464523f55: Generated new TSK 0
I20260928 05:20:40.154989 15928 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260928 05:20:40.221797 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32969
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:20:40.533870 15942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:20:40.534096 15942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:20:40.534205 15942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:20:40.541036 15942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:20:40.541123 15942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:20:40.541262 15942 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 05:20:40.550316 15942 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32969
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:20:40.551388 15942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:20:40.552542 15942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:20:40.564234 15950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:20:40.564316 15947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:20:40.564289 15948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:20:40.565088 15942 server_base.cc:1034] running on GCE node
I20260928 05:20:40.565585 15942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:20:40.566401 15942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:20:40.567595 15942 hybrid_clock.cc:648] HybridClock initialized: now 1790572840567550 us; error 42 us; skew 500 ppm
I20260928 05:20:40.567886 15942 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:20:40 dist-test-slave-whsn krb5kdc[15836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572840, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:20:40.570696 15942 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260928 05:20:40.571969 15942 webserver.cc:460] Webserver started at http://127.0.97.1:41847/ using document root <none> and password file <none>
I20260928 05:20:40.572525 15942 fs_manager.cc:359] Metadata directory not provided
I20260928 05:20:40.572614 15942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:20:40.572878 15942 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:20:40.576257 15942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-0/data/instance:
uuid: "3457ee2e0bfb450cbf6b432ce660d3b9"
format_stamp: "Formatted at 2026-09-28 05:20:40 on dist-test-slave-whsn"
server_key: "caf76cff06d56cf1c8aaa9db349c62a9"
server_key_iv: "a44742f50c5b0d7138129bffb82fa618"
server_key_version: "encryptionkey@0"
I20260928 05:20:40.577198 15942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-0/wal/instance:
uuid: "3457ee2e0bfb450cbf6b432ce660d3b9"
format_stamp: "Formatted at 2026-09-28 05:20:40 on dist-test-slave-whsn"
server_key: "caf76cff06d56cf1c8aaa9db349c62a9"
server_key_iv: "a44742f50c5b0d7138129bffb82fa618"
server_key_version: "encryptionkey@0"
I20260928 05:20:40.583448 15942 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260928 05:20:40.587742 15957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:20:40.589304 15942 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260928 05:20:40.589452 15942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-0/wal
uuid: "3457ee2e0bfb450cbf6b432ce660d3b9"
format_stamp: "Formatted at 2026-09-28 05:20:40 on dist-test-slave-whsn"
server_key: "caf76cff06d56cf1c8aaa9db349c62a9"
server_key_iv: "a44742f50c5b0d7138129bffb82fa618"
server_key_version: "encryptionkey@0"
I20260928 05:20:40.589668 15942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:20:40.615864 15942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:20:40.619822 15942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:20:40.620064 15942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:20:40.620980 15942 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:20:40.622161 15942 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:20:40.622231 15942 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:20:40.622335 15942 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:20:40.622385 15942 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:20:40.651423 15942 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41017
I20260928 05:20:40.651470 16069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41017 every 8 connection(s)
I20260928 05:20:40.652571 15942 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-0/data/info.pb
I20260928 05:20:40.655164 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 15942
I20260928 05:20:40.655356 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-0/wal/instance
I20260928 05:20:40.656158 388 external_mini_cluster.cc:1428] Setting key e0dd46d52cff46dbe28083f11eb64883
Sep 28 05:20:40 dist-test-slave-whsn krb5kdc[15836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572840, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:20:40.671425 16072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:40.654895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55915 (local address 127.0.97.62:32969)
0928 05:20:40.655289 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0928 05:20:40.655294 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:20:40.656377 (+ 1083us) server_negotiation.cc:407] Connection header received
0928 05:20:40.658337 (+ 1960us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:20:40.658343 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:20:40.658421 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:20:40.658514 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:20:40.659885 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:40.660586 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:40.662100 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:40.662491 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:40.662598 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:40.666200 (+ 3602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:20:40.666205 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:20:40.666208 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:20:40.666240 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:20:40.668527 (+ 2287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:40.669366 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:40.669370 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:40.669373 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:40.669499 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:40.669898 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:40.669901 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:40.669903 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:40.670156 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:20:40.670275 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 05:20:40.671015 (+ 740us) server_negotiation.cc:299] Negotiation successful
0928 05:20:40.671173 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":207,"threads_started":1}
I20260928 05:20:40.672649 16073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:40.655477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32969 (local address 127.0.97.1:55915)
0928 05:20:40.656233 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:40.656280 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 05:20:40.658086 (+ 1806us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:20:40.658736 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:20:40.658744 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:20:40.659200 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:20:40.659731 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:20:40.659747 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:20:40.660737 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:20:40.660740 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:20:40.661965 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:20:40.661978 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:20:40.662691 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:20:40.662697 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:20:40.662925 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:40.663566 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:20:40.663587 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:20:40.665988 (+ 2401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:20:40.668645 (+ 2657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:20:40.668652 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:20:40.668667 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:20:40.669216 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:20:40.669597 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:20:40.669602 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:20:40.669606 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:20:40.669809 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:20:40.670272 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:20:40.670278 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:20:40.670675 (+ 397us) client_negotiation.cc:769] Sending connection context
0928 05:20:40.670949 (+ 274us) client_negotiation.cc:240] Negotiation successful
0928 05:20:40.671408 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":270,"threads_started":1}
I20260928 05:20:40.674099 16070 heartbeater.cc:347] Connected to a master server at 127.0.97.62:32969
I20260928 05:20:40.674365 16070 heartbeater.cc:464] Registering TS with master...
I20260928 05:20:40.675019 16070 heartbeater.cc:511] Master 127.0.97.62:32969 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260928 05:20:40.676985 15881 ts_manager.cc:194] Registered new tserver with Master: 3457ee2e0bfb450cbf6b432ce660d3b9 (127.0.97.1:41017)
I20260928 05:20:40.678365 15881 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:55915
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260928 05:20:40.701010 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32969
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:20:41.008560 16076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:20:41.008870 16076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:20:41.009027 16076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:20:41.016080 16076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:20:41.016201 16076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:20:41.016373 16076 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 05:20:41.025823 16076 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32969
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:20:41.027127 16076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:20:41.028438 16076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:20:41.040824 16081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:20:41.040942 16082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:20:41.040915 16084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:20:41.041698 16076 server_base.cc:1034] running on GCE node
I20260928 05:20:41.042235 16076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:20:41.043144 16076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:20:41.044348 16076 hybrid_clock.cc:648] HybridClock initialized: now 1790572841044294 us; error 42 us; skew 500 ppm
I20260928 05:20:41.044636 16076 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:20:41 dist-test-slave-whsn krb5kdc[15836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572841, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:20:41.047760 16076 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260928 05:20:41.049228 16076 webserver.cc:460] Webserver started at http://127.0.97.2:40821/ using document root <none> and password file <none>
I20260928 05:20:41.049856 16076 fs_manager.cc:359] Metadata directory not provided
I20260928 05:20:41.049959 16076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:20:41.050222 16076 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:20:41.053043 16076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-1/data/instance:
uuid: "6f265296eba24246a0acd818360338ef"
format_stamp: "Formatted at 2026-09-28 05:20:41 on dist-test-slave-whsn"
server_key: "ce39ec02e78df8127061b816f18938e2"
server_key_iv: "7c8ac7bd314af6b6bb1a6bc983843871"
server_key_version: "encryptionkey@0"
I20260928 05:20:41.053798 16076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-1/wal/instance:
uuid: "6f265296eba24246a0acd818360338ef"
format_stamp: "Formatted at 2026-09-28 05:20:41 on dist-test-slave-whsn"
server_key: "ce39ec02e78df8127061b816f18938e2"
server_key_iv: "7c8ac7bd314af6b6bb1a6bc983843871"
server_key_version: "encryptionkey@0"
I20260928 05:20:41.058761 16076 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 05:20:41.062808 16091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:20:41.064260 16076 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 05:20:41.064419 16076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-1/wal
uuid: "6f265296eba24246a0acd818360338ef"
format_stamp: "Formatted at 2026-09-28 05:20:41 on dist-test-slave-whsn"
server_key: "ce39ec02e78df8127061b816f18938e2"
server_key_iv: "7c8ac7bd314af6b6bb1a6bc983843871"
server_key_version: "encryptionkey@0"
I20260928 05:20:41.064639 16076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:20:41.081410 16076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:20:41.085383 16076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:20:41.085645 16076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:20:41.086592 16076 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:20:41.088059 16076 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:20:41.088140 16076 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:20:41.088241 16076 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:20:41.088299 16076 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:20:41.119376 16076 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44643
I20260928 05:20:41.119415 16203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44643 every 8 connection(s)
I20260928 05:20:41.120633 16076 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-1/data/info.pb
I20260928 05:20:41.127508 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 16076
I20260928 05:20:41.127664 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-1/wal/instance
I20260928 05:20:41.128131 388 external_mini_cluster.cc:1428] Setting key e413c628cda7d2385a4b923cdba312c8
Sep 28 05:20:41 dist-test-slave-whsn krb5kdc[15836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572841, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:20:41.140606 16072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:41.123180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42023 (local address 127.0.97.62:32969)
0928 05:20:41.123385 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 05:20:41.123391 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:20:41.124479 (+ 1088us) server_negotiation.cc:407] Connection header received
0928 05:20:41.126343 (+ 1864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:20:41.126347 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:20:41.126414 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:20:41.126504 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:20:41.128105 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:41.129056 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:41.131021 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:41.131616 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:41.131763 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:41.135051 (+ 3288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:20:41.135056 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:20:41.135061 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:20:41.135098 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:20:41.137846 (+ 2748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:41.138625 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:41.138629 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:41.138631 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:41.138766 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:41.139179 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:41.139182 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:41.139184 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:41.139420 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:20:41.139524 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 05:20:41.140216 (+ 692us) server_negotiation.cc:299] Negotiation successful
0928 05:20:41.140373 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 05:20:41.141832 16206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:41.123624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32969 (local address 127.0.97.2:42023)
0928 05:20:41.124322 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:41.124366 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 05:20:41.126078 (+ 1712us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:20:41.126730 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:20:41.126738 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:20:41.127285 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:20:41.127931 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:20:41.127957 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:20:41.129237 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:20:41.129242 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:20:41.130874 (+ 1632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:20:41.130887 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:20:41.131819 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:20:41.131825 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:20:41.132020 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:41.132636 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:20:41.132659 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:20:41.134845 (+ 2186us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:20:41.137980 (+ 3135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:20:41.137989 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:20:41.138003 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:20:41.138529 (+ 526us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:20:41.138858 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:20:41.138863 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:20:41.138867 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:20:41.139094 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:20:41.139520 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:20:41.139527 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:20:41.139900 (+ 373us) client_negotiation.cc:769] Sending connection context
0928 05:20:41.140158 (+ 258us) client_negotiation.cc:240] Negotiation successful
0928 05:20:41.140629 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":262,"threads_started":1}
I20260928 05:20:41.143205 16204 heartbeater.cc:347] Connected to a master server at 127.0.97.62:32969
I20260928 05:20:41.143460 16204 heartbeater.cc:464] Registering TS with master...
I20260928 05:20:41.144133 16204 heartbeater.cc:511] Master 127.0.97.62:32969 requested a full tablet report, sending...
I20260928 05:20:41.145471 15881 ts_manager.cc:194] Registered new tserver with Master: 6f265296eba24246a0acd818360338ef (127.0.97.2:44643)
I20260928 05:20:41.146534 15881 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:42023
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260928 05:20:41.172997 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32969
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:20:41.471518 16209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:20:41.471763 16209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:20:41.471851 16209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:20:41.478693 16209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:20:41.478783 16209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:20:41.478896 16209 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 05:20:41.488009 16209 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32969
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:20:41.489161 16209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:20:41.490414 16209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:20:41.502278 16214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:20:41.502341 16215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:20:41.502398 16217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:20:41.503477 16209 server_base.cc:1034] running on GCE node
I20260928 05:20:41.504016 16209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:20:41.504969 16209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:20:41.506196 16209 hybrid_clock.cc:648] HybridClock initialized: now 1790572841506133 us; error 55 us; skew 500 ppm
I20260928 05:20:41.506524 16209 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:20:41 dist-test-slave-whsn krb5kdc[15836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572841, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:20:41.509506 16209 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260928 05:20:41.510879 16209 webserver.cc:460] Webserver started at http://127.0.97.3:44341/ using document root <none> and password file <none>
I20260928 05:20:41.511502 16209 fs_manager.cc:359] Metadata directory not provided
I20260928 05:20:41.511612 16209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:20:41.511902 16209 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:20:41.514644 16209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-2/data/instance:
uuid: "4393522a68d64a33b6a85780a76e8e82"
format_stamp: "Formatted at 2026-09-28 05:20:41 on dist-test-slave-whsn"
server_key: "64e37ebd3b46409260d3eda232db36a9"
server_key_iv: "1a7bae69e865db0cf3ff452037c4e152"
server_key_version: "encryptionkey@0"
I20260928 05:20:41.515342 16209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-2/wal/instance:
uuid: "4393522a68d64a33b6a85780a76e8e82"
format_stamp: "Formatted at 2026-09-28 05:20:41 on dist-test-slave-whsn"
server_key: "64e37ebd3b46409260d3eda232db36a9"
server_key_iv: "1a7bae69e865db0cf3ff452037c4e152"
server_key_version: "encryptionkey@0"
I20260928 05:20:41.520046 16209 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260928 05:20:41.523165 16224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:20:41.524325 16209 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 05:20:41.524461 16209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-2/wal
uuid: "4393522a68d64a33b6a85780a76e8e82"
format_stamp: "Formatted at 2026-09-28 05:20:41 on dist-test-slave-whsn"
server_key: "64e37ebd3b46409260d3eda232db36a9"
server_key_iv: "1a7bae69e865db0cf3ff452037c4e152"
server_key_version: "encryptionkey@0"
I20260928 05:20:41.524618 16209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:20:41.554726 16209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:20:41.558259 16209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:20:41.558511 16209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:20:41.559340 16209 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:20:41.560526 16209 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:20:41.560599 16209 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:20:41.560709 16209 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:20:41.560761 16209 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:20:41.589555 16209 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36609
I20260928 05:20:41.589617 16336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36609 every 8 connection(s)
I20260928 05:20:41.590745 16209 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-2/data/info.pb
I20260928 05:20:41.599823 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 16209
I20260928 05:20:41.599984 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-2/wal/instance
I20260928 05:20:41.600373 388 external_mini_cluster.cc:1428] Setting key 4ec95497116c6ab84af9c78818f11c83
Sep 28 05:20:41 dist-test-slave-whsn krb5kdc[15836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572841, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:20:41.608793 16072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:41.593175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41811 (local address 127.0.97.62:32969)
0928 05:20:41.593364 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 05:20:41.593368 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:20:41.594515 (+ 1147us) server_negotiation.cc:407] Connection header received
0928 05:20:41.596329 (+ 1814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:20:41.596334 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:20:41.596411 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:20:41.596496 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:20:41.597973 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:41.598737 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:41.600257 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:41.600675 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:41.600780 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:41.604118 (+ 3338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:20:41.604122 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:20:41.604126 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:20:41.604156 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:20:41.606400 (+ 2244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:41.606996 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:41.606999 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:41.607002 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:41.607101 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:41.607415 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:41.607417 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:41.607419 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:41.607657 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:20:41.607758 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 05:20:41.608363 (+ 605us) server_negotiation.cc:299] Negotiation successful
0928 05:20:41.608535 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 05:20:41.609520 16339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:41.593657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32969 (local address 127.0.97.3:41811)
0928 05:20:41.594370 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:41.594416 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 05:20:41.596072 (+ 1656us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:20:41.596727 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:20:41.596735 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:20:41.597262 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:20:41.597814 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:20:41.597831 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:20:41.598871 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:20:41.598875 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:20:41.600128 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:20:41.600139 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:20:41.600910 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:20:41.600918 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:20:41.601126 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:41.601781 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:20:41.601802 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:20:41.603937 (+ 2135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:20:41.606523 (+ 2586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:20:41.606529 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:20:41.606540 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:20:41.606910 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:20:41.607170 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:20:41.607173 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:20:41.607175 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:20:41.607341 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:20:41.607742 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:20:41.607747 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:20:41.608014 (+ 267us) client_negotiation.cc:769] Sending connection context
0928 05:20:41.608228 (+ 214us) client_negotiation.cc:240] Negotiation successful
0928 05:20:41.608557 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":280,"threads_started":1}
I20260928 05:20:41.610791 16337 heartbeater.cc:347] Connected to a master server at 127.0.97.62:32969
I20260928 05:20:41.611045 16337 heartbeater.cc:464] Registering TS with master...
I20260928 05:20:41.611666 16337 heartbeater.cc:511] Master 127.0.97.62:32969 requested a full tablet report, sending...
I20260928 05:20:41.613029 15881 ts_manager.cc:194] Registered new tserver with Master: 4393522a68d64a33b6a85780a76e8e82 (127.0.97.3:36609)
I20260928 05:20:41.613823 15881 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:41811
I20260928 05:20:41.615772 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 05:20:41.631011 16072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:41.619977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38832 (local address 127.0.97.62:32969)
0928 05:20:41.620118 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 05:20:41.620122 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:20:41.620529 (+ 407us) server_negotiation.cc:407] Connection header received
0928 05:20:41.620665 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:20:41.620669 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:20:41.620720 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:20:41.620787 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:20:41.622070 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:41.622747 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:41.624027 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:41.624427 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:41.624513 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:41.626526 (+ 2013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:20:41.626531 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:20:41.626534 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:20:41.626567 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:20:41.628919 (+ 2352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:41.629481 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:41.629485 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:41.629487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:41.629611 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:41.629894 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:41.629899 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:41.629902 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:41.630207 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:20:41.630403 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0928 05:20:41.630654 (+ 251us) server_negotiation.cc:299] Negotiation successful
0928 05:20:41.630806 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 05:20:41.634438 15881 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38832:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 05:20:41.640969 15881 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 05:20:41.676935 16352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:41.653565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55934 (local address 127.0.97.1:41017)
0928 05:20:41.668624 (+ 15059us) server_negotiation.cc:206] Beginning negotiation
0928 05:20:41.668630 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:20:41.668654 (+ 24us) server_negotiation.cc:407] Connection header received
0928 05:20:41.668732 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:20:41.668737 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:20:41.668980 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:20:41.669177 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:20:41.669905 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:41.671543 (+ 1638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:41.674103 (+ 2560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:41.675421 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:41.675545 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:41.675766 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0928 05:20:41.676426 (+ 660us) server_negotiation.cc:299] Negotiation successful
0928 05:20:41.676596 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14919,"thread_start_us":15315,"threads_started":1}
I20260928 05:20:41.676959 16351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:41.653198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41017 (local address 127.0.0.1:55934)
0928 05:20:41.666918 (+ 13720us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:41.666940 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 05:20:41.667065 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:20:41.669197 (+ 2132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:20:41.669205 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:20:41.669235 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:20:41.669724 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:20:41.669738 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:20:41.671669 (+ 1931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:20:41.671674 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:20:41.673939 (+ 2265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:20:41.673954 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:20:41.675914 (+ 1960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:20:41.675921 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:20:41.676172 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:41.676235 (+ 63us) client_negotiation.cc:769] Sending connection context
0928 05:20:41.676361 (+ 126us) client_negotiation.cc:240] Negotiation successful
0928 05:20:41.676544 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13585,"mutex_wait_us":26,"thread_start_us":13948,"threads_started":1}
I20260928 05:20:41.678095 16349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:41.651927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36609 (local address 127.0.0.1:56310)
0928 05:20:41.652759 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:41.652799 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 05:20:41.652970 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:20:41.672279 (+ 19309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:20:41.672283 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:20:41.672305 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:20:41.672791 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:20:41.672803 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:20:41.674182 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:20:41.674187 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:20:41.675983 (+ 1796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:20:41.675995 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:20:41.677578 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:20:41.677583 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:20:41.677686 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:41.677708 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 05:20:41.677757 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 05:20:41.677835 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"mutex_wait_us":30,"thread_start_us":508,"threads_started":1}
I20260928 05:20:41.678988 16350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:41.652243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56310 (local address 127.0.97.3:36609)
0928 05:20:41.671699 (+ 19456us) server_negotiation.cc:206] Beginning negotiation
0928 05:20:41.671708 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:20:41.671737 (+ 29us) server_negotiation.cc:407] Connection header received
0928 05:20:41.671833 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:20:41.671840 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:20:41.672095 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:20:41.672280 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:20:41.673002 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:41.674065 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:41.676130 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:41.677381 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:41.678377 (+ 996us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:41.678534 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 05:20:41.678588 (+ 54us) server_negotiation.cc:299] Negotiation successful
0928 05:20:41.678689 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19306,"thread_start_us":19153,"threads_started":1}
I20260928 05:20:41.679231 16353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:41.657122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44643 (local address 127.0.0.1:42002)
0928 05:20:41.671293 (+ 14171us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:41.671309 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 05:20:41.671416 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:20:41.672153 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:20:41.672156 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:20:41.672173 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:20:41.672705 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:20:41.672717 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:20:41.674495 (+ 1778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:20:41.674500 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:20:41.676549 (+ 2049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:20:41.676561 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:20:41.678496 (+ 1935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:20:41.678502 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:20:41.678655 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:41.678686 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 05:20:41.678790 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 05:20:41.678903 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14033,"mutex_wait_us":2,"thread_start_us":13897,"threads_started":1}
I20260928 05:20:41.680132 16354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:41.657616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42002 (local address 127.0.97.2:44643)
0928 05:20:41.670871 (+ 13255us) server_negotiation.cc:206] Beginning negotiation
0928 05:20:41.670880 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:20:41.671526 (+ 646us) server_negotiation.cc:407] Connection header received
0928 05:20:41.671632 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:20:41.671639 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:20:41.671820 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:20:41.671971 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:20:41.672930 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:41.674357 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:41.676720 (+ 2363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:41.678086 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:41.678201 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:41.678367 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 05:20:41.678863 (+ 496us) server_negotiation.cc:299] Negotiation successful
0928 05:20:41.679012 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13109,"thread_start_us":12986,"threads_started":1}
I20260928 05:20:41.681514 16070 heartbeater.cc:503] Master 127.0.97.62:32969 was elected leader, sending a full tablet report...
I20260928 05:20:41.681030 16005 tablet_service.cc:1467] Processing CreateTablet for tablet 4743e73fbcd54d28978dd3809599faa7 (DEFAULT_TABLE table=test-table [id=ad18f2fc366441e598635a0e82e4f88b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:20:41.682516 16005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4743e73fbcd54d28978dd3809599faa7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:20:41.682425 16272 tablet_service.cc:1467] Processing CreateTablet for tablet 4743e73fbcd54d28978dd3809599faa7 (DEFAULT_TABLE table=test-table [id=ad18f2fc366441e598635a0e82e4f88b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:20:41.683772 16272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4743e73fbcd54d28978dd3809599faa7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:20:41.683336 16139 tablet_service.cc:1467] Processing CreateTablet for tablet 4743e73fbcd54d28978dd3809599faa7 (DEFAULT_TABLE table=test-table [id=ad18f2fc366441e598635a0e82e4f88b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:20:41.684564 16139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4743e73fbcd54d28978dd3809599faa7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:20:41.692781 16355 tablet_bootstrap.cc:492] T 4743e73fbcd54d28978dd3809599faa7 P 3457ee2e0bfb450cbf6b432ce660d3b9: Bootstrap starting.
I20260928 05:20:41.695755 16355 tablet_bootstrap.cc:654] T 4743e73fbcd54d28978dd3809599faa7 P 3457ee2e0bfb450cbf6b432ce660d3b9: Neither blocks nor log segments found. Creating new log.
I20260928 05:20:41.697777 16355 log.cc:824] T 4743e73fbcd54d28978dd3809599faa7 P 3457ee2e0bfb450cbf6b432ce660d3b9: Log is configured to *not* fsync() on all Append() calls
I20260928 05:20:41.697856 16356 tablet_bootstrap.cc:492] T 4743e73fbcd54d28978dd3809599faa7 P 4393522a68d64a33b6a85780a76e8e82: Bootstrap starting.
I20260928 05:20:41.700778 16355 tablet_bootstrap.cc:492] T 4743e73fbcd54d28978dd3809599faa7 P 3457ee2e0bfb450cbf6b432ce660d3b9: No bootstrap required, opened a new log
I20260928 05:20:41.701067 16355 ts_tablet_manager.cc:1397] T 4743e73fbcd54d28978dd3809599faa7 P 3457ee2e0bfb450cbf6b432ce660d3b9: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260928 05:20:41.701059 16357 tablet_bootstrap.cc:492] T 4743e73fbcd54d28978dd3809599faa7 P 6f265296eba24246a0acd818360338ef: Bootstrap starting.
I20260928 05:20:41.701160 16356 tablet_bootstrap.cc:654] T 4743e73fbcd54d28978dd3809599faa7 P 4393522a68d64a33b6a85780a76e8e82: Neither blocks nor log segments found. Creating new log.
I20260928 05:20:41.702346 16356 log.cc:824] T 4743e73fbcd54d28978dd3809599faa7 P 4393522a68d64a33b6a85780a76e8e82: Log is configured to *not* fsync() on all Append() calls
I20260928 05:20:41.703949 16357 tablet_bootstrap.cc:654] T 4743e73fbcd54d28978dd3809599faa7 P 6f265296eba24246a0acd818360338ef: Neither blocks nor log segments found. Creating new log.
I20260928 05:20:41.704885 16357 log.cc:824] T 4743e73fbcd54d28978dd3809599faa7 P 6f265296eba24246a0acd818360338ef: Log is configured to *not* fsync() on all Append() calls
I20260928 05:20:41.705233 16356 tablet_bootstrap.cc:492] T 4743e73fbcd54d28978dd3809599faa7 P 4393522a68d64a33b6a85780a76e8e82: No bootstrap required, opened a new log
I20260928 05:20:41.705468 16356 ts_tablet_manager.cc:1397] T 4743e73fbcd54d28978dd3809599faa7 P 4393522a68d64a33b6a85780a76e8e82: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260928 05:20:41.707312 16357 tablet_bootstrap.cc:492] T 4743e73fbcd54d28978dd3809599faa7 P 6f265296eba24246a0acd818360338ef: No bootstrap required, opened a new log
I20260928 05:20:41.707571 16357 ts_tablet_manager.cc:1397] T 4743e73fbcd54d28978dd3809599faa7 P 6f265296eba24246a0acd818360338ef: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.005s
I20260928 05:20:41.710943 16355 raft_consensus.cc:348] T 4743e73fbcd54d28978dd3809599faa7 P 3457ee2e0bfb450cbf6b432ce660d3b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4393522a68d64a33b6a85780a76e8e82" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36609 } } peers { permanent_uuid: "3457ee2e0bfb450cbf6b432ce660d3b9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41017 } } peers { permanent_uuid: "6f265296eba24246a0acd818360338ef" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44643 } }
I20260928 05:20:41.711342 16355 raft_consensus.cc:374] T 4743e73fbcd54d28978dd3809599faa7 P 3457ee2e0bfb450cbf6b432ce660d3b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:20:41.711440 16355 raft_consensus.cc:729] T 4743e73fbcd54d28978dd3809599faa7 P 3457ee2e0bfb450cbf6b432ce660d3b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3457ee2e0bfb450cbf6b432ce660d3b9, State: Initialized, Role: FOLLOWER
I20260928 05:20:41.712041 16355 consensus_queue.cc:260] T 4743e73fbcd54d28978dd3809599faa7 P 3457ee2e0bfb450cbf6b432ce660d3b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4393522a68d64a33b6a85780a76e8e82" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36609 } } peers { permanent_uuid: "3457ee2e0bfb450cbf6b432ce660d3b9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41017 } } peers { permanent_uuid: "6f265296eba24246a0acd818360338ef" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44643 } }
I20260928 05:20:41.715204 16357 raft_consensus.cc:348] T 4743e73fbcd54d28978dd3809599faa7 P 6f265296eba24246a0acd818360338ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4393522a68d64a33b6a85780a76e8e82" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36609 } } peers { permanent_uuid: "3457ee2e0bfb450cbf6b432ce660d3b9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41017 } } peers { permanent_uuid: "6f265296eba24246a0acd818360338ef" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44643 } }
I20260928 05:20:41.715214 16356 raft_consensus.cc:348] T 4743e73fbcd54d28978dd3809599faa7 P 4393522a68d64a33b6a85780a76e8e82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4393522a68d64a33b6a85780a76e8e82" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36609 } } peers { permanent_uuid: "3457ee2e0bfb450cbf6b432ce660d3b9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41017 } } peers { permanent_uuid: "6f265296eba24246a0acd818360338ef" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44643 } }
I20260928 05:20:41.715571 16357 raft_consensus.cc:374] T 4743e73fbcd54d28978dd3809599faa7 P 6f265296eba24246a0acd818360338ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:20:41.715626 16356 raft_consensus.cc:374] T 4743e73fbcd54d28978dd3809599faa7 P 4393522a68d64a33b6a85780a76e8e82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:20:41.715677 16357 raft_consensus.cc:729] T 4743e73fbcd54d28978dd3809599faa7 P 6f265296eba24246a0acd818360338ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f265296eba24246a0acd818360338ef, State: Initialized, Role: FOLLOWER
I20260928 05:20:41.715718 16356 raft_consensus.cc:729] T 4743e73fbcd54d28978dd3809599faa7 P 4393522a68d64a33b6a85780a76e8e82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4393522a68d64a33b6a85780a76e8e82, State: Initialized, Role: FOLLOWER
I20260928 05:20:41.716229 16355 ts_tablet_manager.cc:1428] T 4743e73fbcd54d28978dd3809599faa7 P 3457ee2e0bfb450cbf6b432ce660d3b9: Time spent starting tablet: real 0.015s user 0.012s sys 0.000s
I20260928 05:20:41.716382 16356 consensus_queue.cc:260] T 4743e73fbcd54d28978dd3809599faa7 P 4393522a68d64a33b6a85780a76e8e82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4393522a68d64a33b6a85780a76e8e82" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36609 } } peers { permanent_uuid: "3457ee2e0bfb450cbf6b432ce660d3b9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41017 } } peers { permanent_uuid: "6f265296eba24246a0acd818360338ef" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44643 } }
I20260928 05:20:41.716385 16357 consensus_queue.cc:260] T 4743e73fbcd54d28978dd3809599faa7 P 6f265296eba24246a0acd818360338ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4393522a68d64a33b6a85780a76e8e82" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36609 } } peers { permanent_uuid: "3457ee2e0bfb450cbf6b432ce660d3b9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41017 } } peers { permanent_uuid: "6f265296eba24246a0acd818360338ef" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44643 } }
I20260928 05:20:41.717911 16337 heartbeater.cc:503] Master 127.0.97.62:32969 was elected leader, sending a full tablet report...
I20260928 05:20:41.718161 16357 ts_tablet_manager.cc:1428] T 4743e73fbcd54d28978dd3809599faa7 P 6f265296eba24246a0acd818360338ef: Time spent starting tablet: real 0.010s user 0.005s sys 0.007s
I20260928 05:20:41.718454 16356 ts_tablet_manager.cc:1428] T 4743e73fbcd54d28978dd3809599faa7 P 4393522a68d64a33b6a85780a76e8e82: Time spent starting tablet: real 0.013s user 0.009s sys 0.005s
I20260928 05:20:41.718578 16204 heartbeater.cc:503] Master 127.0.97.62:32969 was elected leader, sending a full tablet report...
I20260928 05:20:41.748759 16363 raft_consensus.cc:482] T 4743e73fbcd54d28978dd3809599faa7 P 4393522a68d64a33b6a85780a76e8e82 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:20:41.749076 16363 raft_consensus.cc:504] T 4743e73fbcd54d28978dd3809599faa7 P 4393522a68d64a33b6a85780a76e8e82 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4393522a68d64a33b6a85780a76e8e82" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36609 } } peers { permanent_uuid: "3457ee2e0bfb450cbf6b432ce660d3b9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41017 } } peers { permanent_uuid: "6f265296eba24246a0acd818360338ef" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44643 } }
I20260928 05:20:41.750607 16363 leader_election.cc:288] T 4743e73fbcd54d28978dd3809599faa7 P 4393522a68d64a33b6a85780a76e8e82 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3457ee2e0bfb450cbf6b432ce660d3b9 (127.0.97.1:41017), 6f265296eba24246a0acd818360338ef (127.0.97.2:44643)
I20260928 05:20:41.756793 16354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:41.751090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46423 (local address 127.0.97.2:44643)
0928 05:20:41.751293 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 05:20:41.751299 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:20:41.751320 (+ 21us) server_negotiation.cc:407] Connection header received
0928 05:20:41.751389 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:20:41.751394 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:20:41.751464 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:20:41.751559 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:20:41.751954 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:41.752935 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:41.755032 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:41.755984 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:41.756113 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:41.756212 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 05:20:41.756444 (+ 232us) server_negotiation.cc:299] Negotiation successful
0928 05:20:41.756560 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 05:20:41.756812 16339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:41.750855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44643 (local address 127.0.97.3:46423)
0928 05:20:41.751063 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:41.751083 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 05:20:41.751199 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:20:41.751597 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:20:41.751606 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:20:41.751632 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:20:41.751828 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:20:41.751838 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:20:41.753043 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:20:41.753047 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:20:41.754888 (+ 1841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:20:41.754903 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:20:41.756129 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:20:41.756135 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:20:41.756305 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:41.756329 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 05:20:41.756406 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 05:20:41.756532 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260928 05:20:41.757267 16352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:41.750920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39493 (local address 127.0.97.1:41017)
0928 05:20:41.751133 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 05:20:41.751140 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:20:41.751738 (+ 598us) server_negotiation.cc:407] Connection header received
0928 05:20:41.751864 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:20:41.751867 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:20:41.751939 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:20:41.752033 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:20:41.752432 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:41.753329 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:41.755442 (+ 2113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:41.756414 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:41.756527 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:41.756619 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 05:20:41.756914 (+ 295us) server_negotiation.cc:299] Negotiation successful
0928 05:20:41.757035 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 05:20:41.757490 16364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:41.750947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41017 (local address 127.0.97.3:39493)
0928 05:20:41.751617 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0928 05:20:41.751636 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 05:20:41.751753 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:20:41.752040 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:20:41.752043 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:20:41.752061 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:20:41.752299 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:20:41.752305 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:20:41.753464 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:20:41.753469 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:20:41.755303 (+ 1834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:20:41.755313 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:20:41.756597 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:20:41.756604 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:20:41.756748 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:41.756770 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 05:20:41.756982 (+ 212us) client_negotiation.cc:240] Negotiation successful
0928 05:20:41.757092 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"mutex_wait_us":3,"thread_start_us":410,"threads_started":1}
I20260928 05:20:41.757763 16159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4743e73fbcd54d28978dd3809599faa7" candidate_uuid: "4393522a68d64a33b6a85780a76e8e82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f265296eba24246a0acd818360338ef" is_pre_election: true
I20260928 05:20:41.758245 16159 raft_consensus.cc:2454] T 4743e73fbcd54d28978dd3809599faa7 P 6f265296eba24246a0acd818360338ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4393522a68d64a33b6a85780a76e8e82 in term 0.
I20260928 05:20:41.758361 16025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4743e73fbcd54d28978dd3809599faa7" candidate_uuid: "4393522a68d64a33b6a85780a76e8e82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3457ee2e0bfb450cbf6b432ce660d3b9" is_pre_election: true
I20260928 05:20:41.758829 16025 raft_consensus.cc:2454] T 4743e73fbcd54d28978dd3809599faa7 P 3457ee2e0bfb450cbf6b432ce660d3b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4393522a68d64a33b6a85780a76e8e82 in term 0.
I20260928 05:20:41.758883 16225 leader_election.cc:302] T 4743e73fbcd54d28978dd3809599faa7 P 4393522a68d64a33b6a85780a76e8e82 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4393522a68d64a33b6a85780a76e8e82, 6f265296eba24246a0acd818360338ef; no voters:
I20260928 05:20:41.759215 16363 raft_consensus.cc:2781] T 4743e73fbcd54d28978dd3809599faa7 P 4393522a68d64a33b6a85780a76e8e82 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 05:20:41.759356 16363 raft_consensus.cc:482] T 4743e73fbcd54d28978dd3809599faa7 P 4393522a68d64a33b6a85780a76e8e82 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 05:20:41.759459 16363 raft_consensus.cc:3037] T 4743e73fbcd54d28978dd3809599faa7 P 4393522a68d64a33b6a85780a76e8e82 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:20:41.761660 16363 raft_consensus.cc:504] T 4743e73fbcd54d28978dd3809599faa7 P 4393522a68d64a33b6a85780a76e8e82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4393522a68d64a33b6a85780a76e8e82" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36609 } } peers { permanent_uuid: "3457ee2e0bfb450cbf6b432ce660d3b9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41017 } } peers { permanent_uuid: "6f265296eba24246a0acd818360338ef" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44643 } }
I20260928 05:20:41.762207 16363 leader_election.cc:288] T 4743e73fbcd54d28978dd3809599faa7 P 4393522a68d64a33b6a85780a76e8e82 [CANDIDATE]: Term 1 election: Requested vote from peers 3457ee2e0bfb450cbf6b432ce660d3b9 (127.0.97.1:41017), 6f265296eba24246a0acd818360338ef (127.0.97.2:44643)
I20260928 05:20:41.762492 16025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4743e73fbcd54d28978dd3809599faa7" candidate_uuid: "4393522a68d64a33b6a85780a76e8e82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3457ee2e0bfb450cbf6b432ce660d3b9"
I20260928 05:20:41.762674 16025 raft_consensus.cc:3037] T 4743e73fbcd54d28978dd3809599faa7 P 3457ee2e0bfb450cbf6b432ce660d3b9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:20:41.762643 16159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4743e73fbcd54d28978dd3809599faa7" candidate_uuid: "4393522a68d64a33b6a85780a76e8e82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f265296eba24246a0acd818360338ef"
I20260928 05:20:41.762827 16159 raft_consensus.cc:3037] T 4743e73fbcd54d28978dd3809599faa7 P 6f265296eba24246a0acd818360338ef [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:20:41.765671 16025 raft_consensus.cc:2454] T 4743e73fbcd54d28978dd3809599faa7 P 3457ee2e0bfb450cbf6b432ce660d3b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4393522a68d64a33b6a85780a76e8e82 in term 1.
I20260928 05:20:41.765686 16159 raft_consensus.cc:2454] T 4743e73fbcd54d28978dd3809599faa7 P 6f265296eba24246a0acd818360338ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4393522a68d64a33b6a85780a76e8e82 in term 1.
I20260928 05:20:41.766038 16227 leader_election.cc:302] T 4743e73fbcd54d28978dd3809599faa7 P 4393522a68d64a33b6a85780a76e8e82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3457ee2e0bfb450cbf6b432ce660d3b9, 4393522a68d64a33b6a85780a76e8e82; no voters:
I20260928 05:20:41.766340 16363 raft_consensus.cc:2781] T 4743e73fbcd54d28978dd3809599faa7 P 4393522a68d64a33b6a85780a76e8e82 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:20:41.766749 16363 raft_consensus.cc:686] T 4743e73fbcd54d28978dd3809599faa7 P 4393522a68d64a33b6a85780a76e8e82 [term 1 LEADER]: Becoming Leader. State: Replica: 4393522a68d64a33b6a85780a76e8e82, State: Running, Role: LEADER
I20260928 05:20:41.767202 16363 consensus_queue.cc:237] T 4743e73fbcd54d28978dd3809599faa7 P 4393522a68d64a33b6a85780a76e8e82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4393522a68d64a33b6a85780a76e8e82" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36609 } } peers { permanent_uuid: "3457ee2e0bfb450cbf6b432ce660d3b9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41017 } } peers { permanent_uuid: "6f265296eba24246a0acd818360338ef" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44643 } }
I20260928 05:20:41.771375 15880 catalog_manager.cc:5495] T 4743e73fbcd54d28978dd3809599faa7 P 4393522a68d64a33b6a85780a76e8e82 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4393522a68d64a33b6a85780a76e8e82 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "4393522a68d64a33b6a85780a76e8e82" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4393522a68d64a33b6a85780a76e8e82" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36609 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3457ee2e0bfb450cbf6b432ce660d3b9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41017 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f265296eba24246a0acd818360338ef" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44643 } health_report { overall_health: UNKNOWN } } }
I20260928 05:20:41.790884 16350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:41.785789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56314 (local address 127.0.97.3:36609)
0928 05:20:41.785970 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 05:20:41.785976 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:20:41.786092 (+ 116us) server_negotiation.cc:407] Connection header received
0928 05:20:41.786301 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:20:41.786305 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:20:41.786372 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:20:41.786500 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:20:41.786846 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:41.787518 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:41.788914 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:41.789304 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:41.789411 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:41.789904 (+ 493us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:20:41.790356 (+ 452us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:20:41.790480 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 05:20:41.790570 (+ 90us) server_negotiation.cc:299] Negotiation successful
0928 05:20:41.790642 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 05:20:41.798616 16159 raft_consensus.cc:1261] T 4743e73fbcd54d28978dd3809599faa7 P 6f265296eba24246a0acd818360338ef [term 1 FOLLOWER]: Refusing update from remote peer 4393522a68d64a33b6a85780a76e8e82: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:20:41.798615 16025 raft_consensus.cc:1261] T 4743e73fbcd54d28978dd3809599faa7 P 3457ee2e0bfb450cbf6b432ce660d3b9 [term 1 FOLLOWER]: Refusing update from remote peer 4393522a68d64a33b6a85780a76e8e82: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:20:41.799559 16365 consensus_queue.cc:1035] T 4743e73fbcd54d28978dd3809599faa7 P 4393522a68d64a33b6a85780a76e8e82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3457ee2e0bfb450cbf6b432ce660d3b9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:20:41.799907 16363 consensus_queue.cc:1035] T 4743e73fbcd54d28978dd3809599faa7 P 4393522a68d64a33b6a85780a76e8e82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f265296eba24246a0acd818360338ef" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:20:41.811551 16369 mvcc.cc:205] Tried to move back new op lower bound from 7334186359997046784 to 7334186359881945088. Current Snapshot: MvccSnapshot[applied={T|T < 7334186359997046784}]
I20260928 05:20:41.813881 16370 mvcc.cc:205] Tried to move back new op lower bound from 7334186359997046784 to 7334186359881945088. Current Snapshot: MvccSnapshot[applied={T|T < 7334186359997046784}]
I20260928 05:20:41.825172 15881 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38832:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:20:41.825402 15881 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38832:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:20:41.828872 15881 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fa08fd07ee1e42f8956f8a8464523f55: Sending DeleteTablet for 3 replicas of tablet 4743e73fbcd54d28978dd3809599faa7
I20260928 05:20:41.830135 16139 tablet_service.cc:1514] Processing DeleteTablet for tablet 4743e73fbcd54d28978dd3809599faa7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:20:41 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42002
I20260928 05:20:41.830265 16272 tablet_service.cc:1514] Processing DeleteTablet for tablet 4743e73fbcd54d28978dd3809599faa7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:20:41 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56310
I20260928 05:20:41.830457 16005 tablet_service.cc:1514] Processing DeleteTablet for tablet 4743e73fbcd54d28978dd3809599faa7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:20:41 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55934
I20260928 05:20:41.831193 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 15942
I20260928 05:20:41.843839 16379 tablet_replica.cc:330] T 4743e73fbcd54d28978dd3809599faa7 P 4393522a68d64a33b6a85780a76e8e82: stopping tablet replica
I20260928 05:20:41.844440 16379 raft_consensus.cc:2229] T 4743e73fbcd54d28978dd3809599faa7 P 4393522a68d64a33b6a85780a76e8e82 [term 1 LEADER]: Raft consensus shutting down.
I20260928 05:20:41.845281 16379 raft_consensus.cc:2258] T 4743e73fbcd54d28978dd3809599faa7 P 4393522a68d64a33b6a85780a76e8e82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:20:41.847440 16379 ts_tablet_manager.cc:1905] T 4743e73fbcd54d28978dd3809599faa7 P 4393522a68d64a33b6a85780a76e8e82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:20:41.848026 16380 tablet_replica.cc:330] T 4743e73fbcd54d28978dd3809599faa7 P 3457ee2e0bfb450cbf6b432ce660d3b9: stopping tablet replica
I20260928 05:20:41.848407 16380 raft_consensus.cc:2229] T 4743e73fbcd54d28978dd3809599faa7 P 3457ee2e0bfb450cbf6b432ce660d3b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:20:41.848902 16380 raft_consensus.cc:2258] T 4743e73fbcd54d28978dd3809599faa7 P 3457ee2e0bfb450cbf6b432ce660d3b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:20:41.850898 16380 ts_tablet_manager.cc:1905] T 4743e73fbcd54d28978dd3809599faa7 P 3457ee2e0bfb450cbf6b432ce660d3b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:20:41.856158 16379 ts_tablet_manager.cc:1918] T 4743e73fbcd54d28978dd3809599faa7 P 4393522a68d64a33b6a85780a76e8e82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:20:41.856382 16379 log.cc:1196] T 4743e73fbcd54d28978dd3809599faa7 P 4393522a68d64a33b6a85780a76e8e82: Deleting WAL directory at /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-2/wal/wals/4743e73fbcd54d28978dd3809599faa7
I20260928 05:20:41.857066 16379 ts_tablet_manager.cc:1939] T 4743e73fbcd54d28978dd3809599faa7 P 4393522a68d64a33b6a85780a76e8e82: Deleting consensus metadata
I20260928 05:20:41.857407 16380 ts_tablet_manager.cc:1918] T 4743e73fbcd54d28978dd3809599faa7 P 3457ee2e0bfb450cbf6b432ce660d3b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:20:41.857556 16380 log.cc:1196] T 4743e73fbcd54d28978dd3809599faa7 P 3457ee2e0bfb450cbf6b432ce660d3b9: Deleting WAL directory at /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-0/wal/wals/4743e73fbcd54d28978dd3809599faa7
I20260928 05:20:41.858059 16380 ts_tablet_manager.cc:1939] T 4743e73fbcd54d28978dd3809599faa7 P 3457ee2e0bfb450cbf6b432ce660d3b9: Deleting consensus metadata
I20260928 05:20:41.859208 15867 catalog_manager.cc:4841] TS 4393522a68d64a33b6a85780a76e8e82 (127.0.97.3:36609): tablet 4743e73fbcd54d28978dd3809599faa7 (table test-table [id=ad18f2fc366441e598635a0e82e4f88b]) successfully deleted
I20260928 05:20:41.859972 16378 tablet_replica.cc:330] T 4743e73fbcd54d28978dd3809599faa7 P 6f265296eba24246a0acd818360338ef: stopping tablet replica
I20260928 05:20:41.860311 15867 catalog_manager.cc:4841] TS 3457ee2e0bfb450cbf6b432ce660d3b9 (127.0.97.1:41017): tablet 4743e73fbcd54d28978dd3809599faa7 (table test-table [id=ad18f2fc366441e598635a0e82e4f88b]) successfully deleted
I20260928 05:20:41.860369 16378 raft_consensus.cc:2229] T 4743e73fbcd54d28978dd3809599faa7 P 6f265296eba24246a0acd818360338ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:20:41.860796 16378 raft_consensus.cc:2258] T 4743e73fbcd54d28978dd3809599faa7 P 6f265296eba24246a0acd818360338ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:20:41.862632 16378 ts_tablet_manager.cc:1905] T 4743e73fbcd54d28978dd3809599faa7 P 6f265296eba24246a0acd818360338ef: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 28 05:20:41 dist-test-slave-whsn krb5kdc[15836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572839, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260928 05:20:41.868705 16378 ts_tablet_manager.cc:1918] T 4743e73fbcd54d28978dd3809599faa7 P 6f265296eba24246a0acd818360338ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:20:41.868834 16378 log.cc:1196] T 4743e73fbcd54d28978dd3809599faa7 P 6f265296eba24246a0acd818360338ef: Deleting WAL directory at /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-1/wal/wals/4743e73fbcd54d28978dd3809599faa7
I20260928 05:20:41.869342 16378 ts_tablet_manager.cc:1939] T 4743e73fbcd54d28978dd3809599faa7 P 6f265296eba24246a0acd818360338ef: Deleting consensus metadata
I20260928 05:20:41.870941 15865 catalog_manager.cc:4841] TS 6f265296eba24246a0acd818360338ef (127.0.97.2:44643): tablet 4743e73fbcd54d28978dd3809599faa7 (table test-table [id=ad18f2fc366441e598635a0e82e4f88b]) successfully deleted
I20260928 05:20:41.874275 16352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:41.832392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55946 (local address 127.0.97.1:41017)
0928 05:20:41.832576 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 05:20:41.832583 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:20:41.857145 (+ 24562us) server_negotiation.cc:407] Connection header received
0928 05:20:41.857448 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:20:41.857454 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:20:41.857535 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:20:41.857614 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:20:41.859578 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:41.860738 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:41.862514 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:41.863166 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:41.863306 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:41.868287 (+ 4981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:20:41.868303 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:20:41.868321 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:20:41.868364 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:20:41.871680 (+ 3316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:41.872339 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:41.872347 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:41.872350 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:41.872479 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:41.872813 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:41.872817 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:41.872819 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:41.873204 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:20:41.873478 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0928 05:20:41.873748 (+ 270us) server_negotiation.cc:299] Negotiation successful
0928 05:20:41.873960 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 05:20:41.874940 16065 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:20:41.961721 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 15942
I20260928 05:20:41.975771 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 16076
Sep 28 05:20:41 dist-test-slave-whsn krb5kdc[15836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572839, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260928 05:20:41.992307 16354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:41.976595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42006 (local address 127.0.97.2:44643)
0928 05:20:41.976837 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0928 05:20:41.976845 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:20:41.976889 (+ 44us) server_negotiation.cc:407] Connection header received
0928 05:20:41.976963 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:20:41.976969 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:20:41.977055 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:20:41.977178 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:20:41.978276 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:41.979426 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:41.980762 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:41.981434 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:41.981579 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:41.985624 (+ 4045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:20:41.985643 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:20:41.985664 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:20:41.985725 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:20:41.989370 (+ 3645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:41.990127 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:41.990138 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:41.990143 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:41.990305 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:41.990621 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:41.990627 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:41.990630 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:41.991126 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:20:41.991429 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0928 05:20:41.991637 (+ 208us) server_negotiation.cc:299] Negotiation successful
0928 05:20:41.991895 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260928 05:20:41.993268 16199 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:20:42.082034 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 16076
I20260928 05:20:42.096763 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 16209
Sep 28 05:20:42 dist-test-slave-whsn krb5kdc[15836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572839, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260928 05:20:42.114066 16350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:20:42.097598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56330 (local address 127.0.97.3:36609)
0928 05:20:42.097766 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 05:20:42.097770 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:20:42.097825 (+ 55us) server_negotiation.cc:407] Connection header received
0928 05:20:42.098071 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:20:42.098076 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:20:42.098166 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:20:42.098280 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:20:42.099993 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:42.100820 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:42.102747 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:20:42.103239 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:20:42.103328 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:20:42.108398 (+ 5070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:20:42.108409 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:20:42.108422 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:20:42.108458 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:20:42.111394 (+ 2936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:42.112076 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:42.112083 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:42.112086 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:42.112255 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:20:42.112584 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:20:42.112587 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:20:42.112589 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:20:42.112953 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:20:42.113242 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0928 05:20:42.113506 (+ 264us) server_negotiation.cc:299] Negotiation successful
0928 05:20:42.113718 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 05:20:42.114675 16332 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:20:42.232430 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 16209
I20260928 05:20:42.249188 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 15849
I20260928 05:20:42.250200 15911 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:20:42.356491 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 15849
I20260928 05:21:02.373833 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:32969
--webserver_interface=127.0.97.62
--webserver_port=36021
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:32969
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:21:02.687172 16386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:21:02.687443 16386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:21:02.687573 16386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:21:02.694548 16386 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 05:21:02.694639 16386 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 05:21:02.694703 16386 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 05:21:02.694766 16386 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:21:02.694837 16386 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:21:02.694932 16386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:21:02.694988 16386 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:21:02.695060 16386 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:21:02.704423 16386 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:32969
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:32969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=36021
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:21:02.705617 16386 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:21:02.706885 16386 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:21:02.727465 16392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:21:02.727442 16395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:21:02.727507 16393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:21:02.727777 16386 server_base.cc:1034] running on GCE node
I20260928 05:21:02.728868 16386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:21:02.730288 16386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:21:02.731534 16386 hybrid_clock.cc:648] HybridClock initialized: now 1790572862731459 us; error 38 us; skew 500 ppm
I20260928 05:21:02.731928 16386 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:21:02 dist-test-slave-whsn krb5kdc[15836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572862, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:21:02.735294 16386 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 05:21:02.736721 16386 webserver.cc:460] Webserver started at http://127.0.97.62:36021/ using document root <none> and password file <none>
I20260928 05:21:02.737478 16386 fs_manager.cc:359] Metadata directory not provided
I20260928 05:21:02.737594 16386 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:21:02.743165 16386 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260928 05:21:02.746044 16402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:21:02.747298 16386 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 05:21:02.747452 16386 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/master-0/wal
uuid: "fa08fd07ee1e42f8956f8a8464523f55"
format_stamp: "Formatted at 2026-09-28 05:20:40 on dist-test-slave-whsn"
server_key: "5283b56f3149cd0cafcc4d4b183d8b08"
server_key_iv: "0e3b1c8c08bb8235dec625a117da870f"
server_key_version: "encryptionkey@0"
I20260928 05:21:02.748431 16386 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:21:02.764017 16386 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:21:02.767916 16386 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:21:02.768172 16386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:21:02.785349 16386 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:32969
I20260928 05:21:02.785389 16453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:32969 every 8 connection(s)
I20260928 05:21:02.786784 16386 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:21:02.793540 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 16386
I20260928 05:21:02.793725 16454 sys_catalog.cc:263] Verifying existing consensus state
I20260928 05:21:02.794490 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:41017
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:32969
--webserver_port=41847
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 05:21:02.798080 16454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa08fd07ee1e42f8956f8a8464523f55: Bootstrap starting.
I20260928 05:21:02.818423 16454 log.cc:824] T 00000000000000000000000000000000 P fa08fd07ee1e42f8956f8a8464523f55: Log is configured to *not* fsync() on all Append() calls
I20260928 05:21:02.830057 16454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa08fd07ee1e42f8956f8a8464523f55: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 05:21:02.830552 16454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa08fd07ee1e42f8956f8a8464523f55: Bootstrap complete.
I20260928 05:21:02.839116 16454 raft_consensus.cc:348] T 00000000000000000000000000000000 P fa08fd07ee1e42f8956f8a8464523f55 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa08fd07ee1e42f8956f8a8464523f55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32969 } }
I20260928 05:21:02.839880 16454 raft_consensus.cc:729] T 00000000000000000000000000000000 P fa08fd07ee1e42f8956f8a8464523f55 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa08fd07ee1e42f8956f8a8464523f55, State: Initialized, Role: FOLLOWER
I20260928 05:21:02.840471 16454 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa08fd07ee1e42f8956f8a8464523f55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa08fd07ee1e42f8956f8a8464523f55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32969 } }
I20260928 05:21:02.840678 16454 raft_consensus.cc:388] T 00000000000000000000000000000000 P fa08fd07ee1e42f8956f8a8464523f55 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:21:02.840788 16454 raft_consensus.cc:482] T 00000000000000000000000000000000 P fa08fd07ee1e42f8956f8a8464523f55 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:21:02.840965 16454 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa08fd07ee1e42f8956f8a8464523f55 [term 1 FOLLOWER]: Advancing to term 2
I20260928 05:21:02.844542 16454 raft_consensus.cc:504] T 00000000000000000000000000000000 P fa08fd07ee1e42f8956f8a8464523f55 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa08fd07ee1e42f8956f8a8464523f55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32969 } }
I20260928 05:21:02.845044 16454 leader_election.cc:302] T 00000000000000000000000000000000 P fa08fd07ee1e42f8956f8a8464523f55 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fa08fd07ee1e42f8956f8a8464523f55; no voters:
I20260928 05:21:02.845607 16454 leader_election.cc:288] T 00000000000000000000000000000000 P fa08fd07ee1e42f8956f8a8464523f55 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 05:21:02.845741 16458 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fa08fd07ee1e42f8956f8a8464523f55 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 05:21:02.846812 16458 raft_consensus.cc:686] T 00000000000000000000000000000000 P fa08fd07ee1e42f8956f8a8464523f55 [term 2 LEADER]: Becoming Leader. State: Replica: fa08fd07ee1e42f8956f8a8464523f55, State: Running, Role: LEADER
I20260928 05:21:02.847391 16458 consensus_queue.cc:237] T 00000000000000000000000000000000 P fa08fd07ee1e42f8956f8a8464523f55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa08fd07ee1e42f8956f8a8464523f55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32969 } }
I20260928 05:21:02.847667 16454 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa08fd07ee1e42f8956f8a8464523f55 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:21:02.849706 16460 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa08fd07ee1e42f8956f8a8464523f55 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa08fd07ee1e42f8956f8a8464523f55. Latest consensus state: current_term: 2 leader_uuid: "fa08fd07ee1e42f8956f8a8464523f55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa08fd07ee1e42f8956f8a8464523f55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32969 } } }
I20260928 05:21:02.850085 16460 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa08fd07ee1e42f8956f8a8464523f55 [sys.catalog]: This master's current role is: LEADER
I20260928 05:21:02.850580 16459 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa08fd07ee1e42f8956f8a8464523f55 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "fa08fd07ee1e42f8956f8a8464523f55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa08fd07ee1e42f8956f8a8464523f55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32969 } } }
I20260928 05:21:02.850862 16459 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa08fd07ee1e42f8956f8a8464523f55 [sys.catalog]: This master's current role is: LEADER
I20260928 05:21:02.855326 16465 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:21:02.863646 16465 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:21:02.865096 16465 catalog_manager.cc:1259] Loaded cluster ID: 267ca691bdf34212ba7e1a115574519b
I20260928 05:21:02.865239 16465 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:21:02.869323 16465 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:21:02.870910 16465 catalog_manager.cc:5879] T 00000000000000000000000000000000 P fa08fd07ee1e42f8956f8a8464523f55: Loaded TSK: 0
I20260928 05:21:02.896919 16475 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 05:21:02.910723 16465 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fa08fd07ee1e42f8956f8a8464523f55: Generated new TSK 1
I20260928 05:21:02.911098 16465 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 05:21:03.153611 16456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:21:03.153882 16456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:21:03.153997 16456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:21:03.161275 16456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:21:03.161377 16456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:21:03.161514 16456 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 05:21:03.171149 16456 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:41017
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=41847
--tserver_master_addrs=127.0.97.62:32969
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:21:03.172386 16456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:21:03.173650 16456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:21:03.187808 16481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:21:03.187821 16482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:21:03.188316 16484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:21:03.190130 16456 server_base.cc:1034] running on GCE node
I20260928 05:21:03.190692 16456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:21:03.191646 16456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:21:03.192878 16456 hybrid_clock.cc:648] HybridClock initialized: now 1790572863192806 us; error 46 us; skew 500 ppm
I20260928 05:21:03.193179 16456 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:21:03 dist-test-slave-whsn krb5kdc[15836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572863, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:21:03.196156 16456 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260928 05:21:03.197722 16456 webserver.cc:460] Webserver started at http://127.0.97.1:41847/ using document root <none> and password file <none>
I20260928 05:21:03.198377 16456 fs_manager.cc:359] Metadata directory not provided
I20260928 05:21:03.198496 16456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:21:03.203433 16456 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 05:21:03.205927 16491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:21:03.207202 16456 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:21:03.207340 16456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-0/wal
uuid: "3457ee2e0bfb450cbf6b432ce660d3b9"
format_stamp: "Formatted at 2026-09-28 05:20:40 on dist-test-slave-whsn"
server_key: "caf76cff06d56cf1c8aaa9db349c62a9"
server_key_iv: "a44742f50c5b0d7138129bffb82fa618"
server_key_version: "encryptionkey@0"
I20260928 05:21:03.208230 16456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:21:03.233553 16456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:21:03.237397 16456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:21:03.237635 16456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:21:03.238647 16456 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:21:03.239986 16456 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:21:03.240080 16456 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:21:03.240180 16456 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:21:03.240248 16456 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:21:03.269963 16456 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41017
I20260928 05:21:03.270035 16603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41017 every 8 connection(s)
I20260928 05:21:03.271310 16456 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-0/data/info.pb
I20260928 05:21:03.278314 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 16456
I20260928 05:21:03.278887 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:44643
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:32969
--webserver_port=40821
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 05:21:03 dist-test-slave-whsn krb5kdc[15836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572863, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:21:03.292662 16606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:03.273958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42033 (local address 127.0.97.62:32969)
0928 05:21:03.274532 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0928 05:21:03.274543 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 05:21:03.275162 (+ 619us) server_negotiation.cc:407] Connection header received
0928 05:21:03.277186 (+ 2024us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:21:03.277200 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:21:03.277520 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:21:03.277843 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:21:03.279144 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:03.280347 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:03.281872 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:03.282498 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:03.282657 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:03.285848 (+ 3191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:21:03.285858 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:21:03.285871 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:21:03.285907 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:21:03.288538 (+ 2631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:21:03.289302 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:21:03.289309 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:21:03.289314 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:21:03.289465 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:21:03.289847 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:21:03.289851 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:21:03.289853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:21:03.290275 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:21:03.290441 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 05:21:03.291171 (+ 730us) server_negotiation.cc:299] Negotiation successful
0928 05:21:03.291522 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":229,"threads_started":1}
I20260928 05:21:03.292663 16607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:03.274193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32969 (local address 127.0.97.1:42033)
0928 05:21:03.274985 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:03.275048 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0928 05:21:03.276720 (+ 1672us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:21:03.277920 (+ 1200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:21:03.277927 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:21:03.278416 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:21:03.278926 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:03.278942 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:03.280511 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:03.280515 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:03.281741 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:03.281754 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:03.282709 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:03.282716 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:03.282952 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:03.283565 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:21:03.283585 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:21:03.285611 (+ 2026us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:21:03.288663 (+ 3052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:21:03.288671 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:21:03.288684 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:21:03.289173 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:21:03.289568 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:21:03.289573 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:21:03.289577 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:21:03.289740 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:21:03.290408 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:21:03.290415 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:21:03.290710 (+ 295us) client_negotiation.cc:769] Sending connection context
0928 05:21:03.290925 (+ 215us) client_negotiation.cc:240] Negotiation successful
0928 05:21:03.291418 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":316,"threads_started":1}
I20260928 05:21:03.295293 16604 heartbeater.cc:347] Connected to a master server at 127.0.97.62:32969
I20260928 05:21:03.295553 16604 heartbeater.cc:464] Registering TS with master...
I20260928 05:21:03.296197 16604 heartbeater.cc:511] Master 127.0.97.62:32969 requested a full tablet report, sending...
I20260928 05:21:03.298141 16419 ts_manager.cc:194] Registered new tserver with Master: 3457ee2e0bfb450cbf6b432ce660d3b9 (127.0.97.1:41017)
I20260928 05:21:03.299571 16419 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:42033
W20260928 05:21:03.584210 16608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:21:03.584458 16608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:21:03.584579 16608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:21:03.591723 16608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:21:03.591838 16608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:21:03.591986 16608 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 05:21:03.601119 16608 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:44643
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=40821
--tserver_master_addrs=127.0.97.62:32969
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:21:03.602223 16608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:21:03.603423 16608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:21:03.615103 16613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:21:03.615126 16614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:21:03.615262 16616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:21:03.615955 16608 server_base.cc:1034] running on GCE node
I20260928 05:21:03.616466 16608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:21:03.617326 16608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:21:03.618500 16608 hybrid_clock.cc:648] HybridClock initialized: now 1790572863618450 us; error 45 us; skew 500 ppm
I20260928 05:21:03.618745 16608 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:21:03 dist-test-slave-whsn krb5kdc[15836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572863, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:21:03.621510 16608 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260928 05:21:03.622704 16608 webserver.cc:460] Webserver started at http://127.0.97.2:40821/ using document root <none> and password file <none>
I20260928 05:21:03.623283 16608 fs_manager.cc:359] Metadata directory not provided
I20260928 05:21:03.623384 16608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:21:03.628100 16608 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.001s
I20260928 05:21:03.630483 16623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:21:03.631644 16608 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 05:21:03.631783 16608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-1/wal
uuid: "6f265296eba24246a0acd818360338ef"
format_stamp: "Formatted at 2026-09-28 05:20:41 on dist-test-slave-whsn"
server_key: "ce39ec02e78df8127061b816f18938e2"
server_key_iv: "7c8ac7bd314af6b6bb1a6bc983843871"
server_key_version: "encryptionkey@0"
I20260928 05:21:03.632658 16608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:21:03.644896 16608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:21:03.648265 16608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:21:03.648486 16608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:21:03.649403 16608 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:21:03.650532 16608 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:21:03.650628 16608 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:21:03.650722 16608 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:21:03.650789 16608 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:21:03.678718 16608 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44643
I20260928 05:21:03.678776 16735 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44643 every 8 connection(s)
I20260928 05:21:03.680034 16608 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-1/data/info.pb
I20260928 05:21:03.688117 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 16608
I20260928 05:21:03.688657 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:36609
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:32969
--webserver_port=44341
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 05:21:03 dist-test-slave-whsn krb5kdc[15836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572863, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:21:03.698346 16606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:03.682318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38205 (local address 127.0.97.62:32969)
0928 05:21:03.682487 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 05:21:03.682491 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:21:03.683518 (+ 1027us) server_negotiation.cc:407] Connection header received
0928 05:21:03.685307 (+ 1789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:21:03.685312 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:21:03.685384 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:21:03.685458 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:21:03.686783 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:03.687446 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:03.688945 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:03.689545 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:03.689664 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:03.693059 (+ 3395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:21:03.693065 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:21:03.693069 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:21:03.693105 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:21:03.695619 (+ 2514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:21:03.696271 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:21:03.696275 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:21:03.696278 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:21:03.696389 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:21:03.696821 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:21:03.696825 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:21:03.696827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:21:03.697128 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:21:03.697242 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 05:21:03.697940 (+ 698us) server_negotiation.cc:299] Negotiation successful
0928 05:21:03.698106 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 05:21:03.699533 16738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:03.682721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32969 (local address 127.0.97.2:38205)
0928 05:21:03.683366 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:03.683406 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 05:21:03.685051 (+ 1645us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:21:03.685645 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:21:03.685653 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:21:03.686111 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:21:03.686651 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:03.686668 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:03.687565 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:03.687568 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:03.688780 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:03.688793 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:03.689729 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:03.689733 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:03.689920 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:03.690498 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:21:03.690528 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:21:03.692811 (+ 2283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:21:03.695739 (+ 2928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:21:03.695747 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:21:03.695762 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:21:03.696155 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:21:03.696488 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:21:03.696493 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:21:03.696496 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:21:03.696702 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:21:03.697242 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:21:03.697249 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:21:03.697601 (+ 352us) client_negotiation.cc:769] Sending connection context
0928 05:21:03.697855 (+ 254us) client_negotiation.cc:240] Negotiation successful
0928 05:21:03.698321 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":252,"threads_started":1}
I20260928 05:21:03.700839 16736 heartbeater.cc:347] Connected to a master server at 127.0.97.62:32969
I20260928 05:21:03.701157 16736 heartbeater.cc:464] Registering TS with master...
I20260928 05:21:03.701812 16736 heartbeater.cc:511] Master 127.0.97.62:32969 requested a full tablet report, sending...
I20260928 05:21:03.703079 16419 ts_manager.cc:194] Registered new tserver with Master: 6f265296eba24246a0acd818360338ef (127.0.97.2:44643)
I20260928 05:21:03.703811 16419 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:38205
W20260928 05:21:04.007114 16739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:21:04.007391 16739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:21:04.007514 16739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:21:04.014652 16739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:21:04.014757 16739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:21:04.014909 16739 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 05:21:04.024421 16739 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:36609
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=44341
--tserver_master_addrs=127.0.97.62:32969
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:21:04.025662 16739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:21:04.026906 16739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:21:04.039582 16747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:21:04.039669 16744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:21:04.039680 16745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:21:04.045873 16739 server_base.cc:1034] running on GCE node
I20260928 05:21:04.046536 16739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:21:04.047483 16739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:21:04.048677 16739 hybrid_clock.cc:648] HybridClock initialized: now 1790572864048634 us; error 32 us; skew 500 ppm
I20260928 05:21:04.049043 16739 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:21:04 dist-test-slave-whsn krb5kdc[15836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572864, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:21:04.052050 16739 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260928 05:21:04.053530 16739 webserver.cc:460] Webserver started at http://127.0.97.3:44341/ using document root <none> and password file <none>
I20260928 05:21:04.054211 16739 fs_manager.cc:359] Metadata directory not provided
I20260928 05:21:04.054334 16739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:21:04.059520 16739 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 05:21:04.062175 16754 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:21:04.063450 16739 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:21:04.063599 16739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-2/wal
uuid: "4393522a68d64a33b6a85780a76e8e82"
format_stamp: "Formatted at 2026-09-28 05:20:41 on dist-test-slave-whsn"
server_key: "64e37ebd3b46409260d3eda232db36a9"
server_key_iv: "1a7bae69e865db0cf3ff452037c4e152"
server_key_version: "encryptionkey@0"
I20260928 05:21:04.064527 16739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:21:04.082204 16739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:21:04.086114 16739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:21:04.086354 16739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:21:04.087268 16739 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:21:04.088488 16739 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:21:04.088575 16739 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:21:04.088675 16739 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:21:04.088740 16739 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:21:04.118217 16739 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36609
I20260928 05:21:04.118285 16866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36609 every 8 connection(s)
I20260928 05:21:04.119575 16739 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-2/data/info.pb
I20260928 05:21:04.122236 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 16739
Sep 28 05:21:04 dist-test-slave-whsn krb5kdc[15836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572864, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:21:04.138430 16871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:04.123175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40510 (local address 127.0.97.62:32969)
0928 05:21:04.124843 (+ 1668us) server_negotiation.cc:206] Beginning negotiation
0928 05:21:04.124847 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:21:04.124883 (+ 36us) server_negotiation.cc:407] Connection header received
0928 05:21:04.124940 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:21:04.124944 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:21:04.125021 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:21:04.125110 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:21:04.127055 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:04.127826 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:04.129674 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:04.130300 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:04.130418 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:04.132982 (+ 2564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:21:04.132987 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:21:04.132990 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:21:04.133027 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:21:04.135880 (+ 2853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:21:04.136450 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:21:04.136456 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:21:04.136460 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:21:04.136577 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:21:04.137015 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:21:04.137021 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:21:04.137024 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:21:04.137309 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:21:04.137575 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0928 05:21:04.137840 (+ 265us) server_negotiation.cc:299] Negotiation successful
0928 05:21:04.138052 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1583,"thread_start_us":1483,"threads_started":1}
I20260928 05:21:04.140491 16606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:04.122012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48475 (local address 127.0.97.62:32969)
0928 05:21:04.122213 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 05:21:04.122220 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:21:04.124173 (+ 1953us) server_negotiation.cc:407] Connection header received
0928 05:21:04.126316 (+ 2143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:21:04.126319 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:21:04.126360 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:21:04.126415 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:21:04.128029 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:04.129129 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:04.131018 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:04.131439 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:04.131549 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:04.135089 (+ 3540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:21:04.135097 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:21:04.135100 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:21:04.135142 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:21:04.137670 (+ 2528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:21:04.138291 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:21:04.138296 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:21:04.138299 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:21:04.138457 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:21:04.138950 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:21:04.138956 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:21:04.138959 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:21:04.139251 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:21:04.139387 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 05:21:04.140110 (+ 723us) server_negotiation.cc:299] Negotiation successful
0928 05:21:04.140245 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 05:21:04.141759 16869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:04.122505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32969 (local address 127.0.97.3:48475)
0928 05:21:04.124045 (+ 1540us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:04.124079 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 05:21:04.126064 (+ 1985us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:21:04.126671 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:21:04.126681 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:21:04.127274 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:21:04.127838 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:04.127861 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:04.129269 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:04.129272 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:04.130875 (+ 1603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:04.130890 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:04.131646 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:04.131654 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:04.131898 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:04.132469 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:21:04.132489 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:21:04.134898 (+ 2409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:21:04.137787 (+ 2889us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:21:04.137795 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:21:04.137811 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:21:04.138189 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:21:04.138609 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:21:04.138614 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:21:04.138617 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:21:04.138857 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:21:04.139369 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:21:04.139375 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:21:04.139730 (+ 355us) client_negotiation.cc:769] Sending connection context
0928 05:21:04.140040 (+ 310us) client_negotiation.cc:240] Negotiation successful
0928 05:21:04.140530 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1249,"thread_start_us":1054,"threads_started":1}
I20260928 05:21:04.143045 16867 heartbeater.cc:347] Connected to a master server at 127.0.97.62:32969
I20260928 05:21:04.143333 16867 heartbeater.cc:464] Registering TS with master...
I20260928 05:21:04.144038 16867 heartbeater.cc:511] Master 127.0.97.62:32969 requested a full tablet report, sending...
I20260928 05:21:04.145363 16419 ts_manager.cc:194] Registered new tserver with Master: 4393522a68d64a33b6a85780a76e8e82 (127.0.97.3:36609)
I20260928 05:21:04.146107 16419 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:48475
I20260928 05:21:04.152961 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 05:21:04.159091 16606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:04.153771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40524 (local address 127.0.97.62:32969)
0928 05:21:04.153945 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 05:21:04.153951 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:21:04.154378 (+ 427us) server_negotiation.cc:407] Connection header received
0928 05:21:04.154588 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:21:04.154591 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:21:04.154712 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:21:04.154856 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:21:04.155261 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:04.155960 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:04.157458 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:04.157867 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:04.157961 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:04.158381 (+ 420us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:21:04.158629 (+ 248us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0928 05:21:04.158892 (+ 263us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40524: token expired
Metrics: {"server-negotiator.queue_time_us":74}
W20260928 05:21:04.159215 16606 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40524: token expired
W20260928 05:21:04.159723 16340 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.97.62:32969)
I20260928 05:21:04.170993 16606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:04.160104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40526 (local address 127.0.97.62:32969)
0928 05:21:04.160246 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 05:21:04.160249 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 05:21:04.160264 (+ 15us) server_negotiation.cc:407] Connection header received
0928 05:21:04.160373 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:21:04.160376 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:21:04.160438 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:21:04.160521 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:21:04.161776 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:04.162442 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:04.163622 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:04.163987 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:04.164071 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:04.166399 (+ 2328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:21:04.166403 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:21:04.166406 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:21:04.166432 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:21:04.168840 (+ 2408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:21:04.169469 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:21:04.169472 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:21:04.169475 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:21:04.169595 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:21:04.169872 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:21:04.169875 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:21:04.169877 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:21:04.170158 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:21:04.170330 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0928 05:21:04.170575 (+ 245us) server_negotiation.cc:299] Negotiation successful
0928 05:21:04.170729 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 05:21:04.173983 16419 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40526:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 05:21:04.180925 16419 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 05:21:04.203976 16878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:04.191565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44643 (local address 127.0.0.1:47244)
0928 05:21:04.192972 (+ 1407us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:04.193021 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 05:21:04.193250 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:21:04.194528 (+ 1278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:21:04.194536 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:21:04.194568 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:21:04.195302 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:04.195316 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:04.197701 (+ 2385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:04.197707 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:04.200567 (+ 2860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:04.200587 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:04.203048 (+ 2461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:04.203054 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:04.203263 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:04.203320 (+ 57us) client_negotiation.cc:769] Sending connection context
0928 05:21:04.203421 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 05:21:04.203627 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":944,"threads_started":1}
I20260928 05:21:04.204408 16880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:04.192073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47244 (local address 127.0.97.2:44643)
0928 05:21:04.193844 (+ 1771us) server_negotiation.cc:206] Beginning negotiation
0928 05:21:04.193853 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:21:04.193888 (+ 35us) server_negotiation.cc:407] Connection header received
0928 05:21:04.193991 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:21:04.194000 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:21:04.194270 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:21:04.195547 (+ 1277us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:21:04.195837 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:04.197536 (+ 1699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:04.200785 (+ 3249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:04.202371 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:04.203684 (+ 1313us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:04.203877 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0928 05:21:04.203945 (+ 68us) server_negotiation.cc:299] Negotiation successful
0928 05:21:04.204117 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1586,"thread_start_us":1439,"threads_started":1}
I20260928 05:21:04.208703 16671 tablet_service.cc:1467] Processing CreateTablet for tablet 196ffebfe3f548ff8bf8674ab4773f25 (DEFAULT_TABLE table=test-table [id=8cc077ac2093419e80bf1bc4d5df2f3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:21:04.210180 16671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 196ffebfe3f548ff8bf8674ab4773f25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:21:04.211537 16878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:04.191833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41017 (local address 127.0.0.1:54982)
0928 05:21:04.204183 (+ 12350us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:04.204202 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 05:21:04.204307 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:21:04.205066 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:21:04.205070 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:21:04.205112 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:21:04.205264 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:04.205272 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:04.207068 (+ 1796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:04.207073 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:04.208918 (+ 1845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:04.208933 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:04.210637 (+ 1704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:04.210643 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:04.210928 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:04.210961 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 05:21:04.211030 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 05:21:04.211183 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12255,"mutex_wait_us":19,"spinlock_wait_cycles":18048,"thread_start_us":531,"threads_started":1}
I20260928 05:21:04.211917 16879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:04.192038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54982 (local address 127.0.97.1:41017)
0928 05:21:04.202848 (+ 10810us) server_negotiation.cc:206] Beginning negotiation
0928 05:21:04.202856 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:21:04.204429 (+ 1573us) server_negotiation.cc:407] Connection header received
0928 05:21:04.204549 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:21:04.204557 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:21:04.204839 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:21:04.205394 (+ 555us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:21:04.205579 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:04.206919 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:04.209109 (+ 2190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:04.210453 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:04.211143 (+ 690us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:04.211329 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0928 05:21:04.211393 (+ 64us) server_negotiation.cc:299] Negotiation successful
0928 05:21:04.211558 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10645,"thread_start_us":11351,"threads_started":1}
I20260928 05:21:04.215260 16877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:04.193181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36609 (local address 127.0.0.1:48536)
0928 05:21:04.204986 (+ 11805us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:04.205013 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 05:21:04.205115 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:21:04.207345 (+ 2230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:21:04.207350 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:21:04.207376 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:21:04.207800 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:04.207812 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:04.210205 (+ 2393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:04.210210 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:04.212615 (+ 2405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:04.212627 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:04.214451 (+ 1824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:04.214457 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:04.214634 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:04.214668 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 05:21:04.214762 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 05:21:04.214890 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11654,"mutex_wait_us":62,"spinlock_wait_cycles":7296,"thread_start_us":16400,"threads_started":1}
I20260928 05:21:04.215294 16882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:04.193742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48536 (local address 127.0.97.3:36609)
0928 05:21:04.206433 (+ 12691us) server_negotiation.cc:206] Beginning negotiation
0928 05:21:04.206442 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:21:04.206473 (+ 31us) server_negotiation.cc:407] Connection header received
0928 05:21:04.206583 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:21:04.206591 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:21:04.206812 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:21:04.207073 (+ 261us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:21:04.208068 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:04.210075 (+ 2007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:04.212776 (+ 2701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:04.214224 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:04.214358 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:04.214572 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0928 05:21:04.214814 (+ 242us) server_negotiation.cc:299] Negotiation successful
0928 05:21:04.214985 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12527,"thread_start_us":12370,"threads_started":1}
I20260928 05:21:04.215667 16539 tablet_service.cc:1467] Processing CreateTablet for tablet 196ffebfe3f548ff8bf8674ab4773f25 (DEFAULT_TABLE table=test-table [id=8cc077ac2093419e80bf1bc4d5df2f3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:21:04.216902 16539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 196ffebfe3f548ff8bf8674ab4773f25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:21:04.219131 16802 tablet_service.cc:1467] Processing CreateTablet for tablet 196ffebfe3f548ff8bf8674ab4773f25 (DEFAULT_TABLE table=test-table [id=8cc077ac2093419e80bf1bc4d5df2f3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:21:04.220458 16802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 196ffebfe3f548ff8bf8674ab4773f25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:21:04.223170 16883 tablet_bootstrap.cc:492] T 196ffebfe3f548ff8bf8674ab4773f25 P 6f265296eba24246a0acd818360338ef: Bootstrap starting.
I20260928 05:21:04.226730 16883 tablet_bootstrap.cc:654] T 196ffebfe3f548ff8bf8674ab4773f25 P 6f265296eba24246a0acd818360338ef: Neither blocks nor log segments found. Creating new log.
I20260928 05:21:04.227669 16883 log.cc:824] T 196ffebfe3f548ff8bf8674ab4773f25 P 6f265296eba24246a0acd818360338ef: Log is configured to *not* fsync() on all Append() calls
I20260928 05:21:04.229974 16883 tablet_bootstrap.cc:492] T 196ffebfe3f548ff8bf8674ab4773f25 P 6f265296eba24246a0acd818360338ef: No bootstrap required, opened a new log
I20260928 05:21:04.230206 16883 ts_tablet_manager.cc:1397] T 196ffebfe3f548ff8bf8674ab4773f25 P 6f265296eba24246a0acd818360338ef: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260928 05:21:04.231515 16885 tablet_bootstrap.cc:492] T 196ffebfe3f548ff8bf8674ab4773f25 P 3457ee2e0bfb450cbf6b432ce660d3b9: Bootstrap starting.
I20260928 05:21:04.234903 16885 tablet_bootstrap.cc:654] T 196ffebfe3f548ff8bf8674ab4773f25 P 3457ee2e0bfb450cbf6b432ce660d3b9: Neither blocks nor log segments found. Creating new log.
I20260928 05:21:04.236071 16885 log.cc:824] T 196ffebfe3f548ff8bf8674ab4773f25 P 3457ee2e0bfb450cbf6b432ce660d3b9: Log is configured to *not* fsync() on all Append() calls
I20260928 05:21:04.237357 16886 tablet_bootstrap.cc:492] T 196ffebfe3f548ff8bf8674ab4773f25 P 4393522a68d64a33b6a85780a76e8e82: Bootstrap starting.
I20260928 05:21:04.239059 16885 tablet_bootstrap.cc:492] T 196ffebfe3f548ff8bf8674ab4773f25 P 3457ee2e0bfb450cbf6b432ce660d3b9: No bootstrap required, opened a new log
I20260928 05:21:04.239311 16885 ts_tablet_manager.cc:1397] T 196ffebfe3f548ff8bf8674ab4773f25 P 3457ee2e0bfb450cbf6b432ce660d3b9: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260928 05:21:04.239683 16883 raft_consensus.cc:348] T 196ffebfe3f548ff8bf8674ab4773f25 P 6f265296eba24246a0acd818360338ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f265296eba24246a0acd818360338ef" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44643 } } peers { permanent_uuid: "3457ee2e0bfb450cbf6b432ce660d3b9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41017 } } peers { permanent_uuid: "4393522a68d64a33b6a85780a76e8e82" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36609 } }
I20260928 05:21:04.240093 16883 raft_consensus.cc:374] T 196ffebfe3f548ff8bf8674ab4773f25 P 6f265296eba24246a0acd818360338ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:21:04.240190 16883 raft_consensus.cc:729] T 196ffebfe3f548ff8bf8674ab4773f25 P 6f265296eba24246a0acd818360338ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f265296eba24246a0acd818360338ef, State: Initialized, Role: FOLLOWER
I20260928 05:21:04.240772 16883 consensus_queue.cc:260] T 196ffebfe3f548ff8bf8674ab4773f25 P 6f265296eba24246a0acd818360338ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f265296eba24246a0acd818360338ef" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44643 } } peers { permanent_uuid: "3457ee2e0bfb450cbf6b432ce660d3b9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41017 } } peers { permanent_uuid: "4393522a68d64a33b6a85780a76e8e82" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36609 } }
I20260928 05:21:04.241856 16736 heartbeater.cc:503] Master 127.0.97.62:32969 was elected leader, sending a full tablet report...
I20260928 05:21:04.242002 16886 tablet_bootstrap.cc:654] T 196ffebfe3f548ff8bf8674ab4773f25 P 4393522a68d64a33b6a85780a76e8e82: Neither blocks nor log segments found. Creating new log.
I20260928 05:21:04.242396 16883 ts_tablet_manager.cc:1428] T 196ffebfe3f548ff8bf8674ab4773f25 P 6f265296eba24246a0acd818360338ef: Time spent starting tablet: real 0.012s user 0.009s sys 0.004s
I20260928 05:21:04.243108 16886 log.cc:824] T 196ffebfe3f548ff8bf8674ab4773f25 P 4393522a68d64a33b6a85780a76e8e82: Log is configured to *not* fsync() on all Append() calls
I20260928 05:21:04.245911 16886 tablet_bootstrap.cc:492] T 196ffebfe3f548ff8bf8674ab4773f25 P 4393522a68d64a33b6a85780a76e8e82: No bootstrap required, opened a new log
I20260928 05:21:04.246165 16886 ts_tablet_manager.cc:1397] T 196ffebfe3f548ff8bf8674ab4773f25 P 4393522a68d64a33b6a85780a76e8e82: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.002s
I20260928 05:21:04.249048 16885 raft_consensus.cc:348] T 196ffebfe3f548ff8bf8674ab4773f25 P 3457ee2e0bfb450cbf6b432ce660d3b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f265296eba24246a0acd818360338ef" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44643 } } peers { permanent_uuid: "3457ee2e0bfb450cbf6b432ce660d3b9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41017 } } peers { permanent_uuid: "4393522a68d64a33b6a85780a76e8e82" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36609 } }
I20260928 05:21:04.249451 16885 raft_consensus.cc:374] T 196ffebfe3f548ff8bf8674ab4773f25 P 3457ee2e0bfb450cbf6b432ce660d3b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:21:04.249543 16885 raft_consensus.cc:729] T 196ffebfe3f548ff8bf8674ab4773f25 P 3457ee2e0bfb450cbf6b432ce660d3b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3457ee2e0bfb450cbf6b432ce660d3b9, State: Initialized, Role: FOLLOWER
I20260928 05:21:04.250142 16885 consensus_queue.cc:260] T 196ffebfe3f548ff8bf8674ab4773f25 P 3457ee2e0bfb450cbf6b432ce660d3b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f265296eba24246a0acd818360338ef" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44643 } } peers { permanent_uuid: "3457ee2e0bfb450cbf6b432ce660d3b9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41017 } } peers { permanent_uuid: "4393522a68d64a33b6a85780a76e8e82" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36609 } }
I20260928 05:21:04.251124 16604 heartbeater.cc:503] Master 127.0.97.62:32969 was elected leader, sending a full tablet report...
I20260928 05:21:04.251819 16885 ts_tablet_manager.cc:1428] T 196ffebfe3f548ff8bf8674ab4773f25 P 3457ee2e0bfb450cbf6b432ce660d3b9: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260928 05:21:04.254873 16886 raft_consensus.cc:348] T 196ffebfe3f548ff8bf8674ab4773f25 P 4393522a68d64a33b6a85780a76e8e82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f265296eba24246a0acd818360338ef" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44643 } } peers { permanent_uuid: "3457ee2e0bfb450cbf6b432ce660d3b9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41017 } } peers { permanent_uuid: "4393522a68d64a33b6a85780a76e8e82" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36609 } }
I20260928 05:21:04.255193 16886 raft_consensus.cc:374] T 196ffebfe3f548ff8bf8674ab4773f25 P 4393522a68d64a33b6a85780a76e8e82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:21:04.255265 16886 raft_consensus.cc:729] T 196ffebfe3f548ff8bf8674ab4773f25 P 4393522a68d64a33b6a85780a76e8e82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4393522a68d64a33b6a85780a76e8e82, State: Initialized, Role: FOLLOWER
I20260928 05:21:04.255712 16886 consensus_queue.cc:260] T 196ffebfe3f548ff8bf8674ab4773f25 P 4393522a68d64a33b6a85780a76e8e82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f265296eba24246a0acd818360338ef" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44643 } } peers { permanent_uuid: "3457ee2e0bfb450cbf6b432ce660d3b9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41017 } } peers { permanent_uuid: "4393522a68d64a33b6a85780a76e8e82" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36609 } }
I20260928 05:21:04.256577 16867 heartbeater.cc:503] Master 127.0.97.62:32969 was elected leader, sending a full tablet report...
I20260928 05:21:04.257174 16886 ts_tablet_manager.cc:1428] T 196ffebfe3f548ff8bf8674ab4773f25 P 4393522a68d64a33b6a85780a76e8e82: Time spent starting tablet: real 0.011s user 0.011s sys 0.001s
W20260928 05:21:04.273226 16605 tablet.cc:2369] T 196ffebfe3f548ff8bf8674ab4773f25 P 3457ee2e0bfb450cbf6b432ce660d3b9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 05:21:04.359416 16888 raft_consensus.cc:482] T 196ffebfe3f548ff8bf8674ab4773f25 P 6f265296eba24246a0acd818360338ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:21:04.359738 16888 raft_consensus.cc:504] T 196ffebfe3f548ff8bf8674ab4773f25 P 6f265296eba24246a0acd818360338ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f265296eba24246a0acd818360338ef" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44643 } } peers { permanent_uuid: "3457ee2e0bfb450cbf6b432ce660d3b9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41017 } } peers { permanent_uuid: "4393522a68d64a33b6a85780a76e8e82" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36609 } }
I20260928 05:21:04.361377 16888 leader_election.cc:288] T 196ffebfe3f548ff8bf8674ab4773f25 P 6f265296eba24246a0acd818360338ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3457ee2e0bfb450cbf6b432ce660d3b9 (127.0.97.1:41017), 4393522a68d64a33b6a85780a76e8e82 (127.0.97.3:36609)
I20260928 05:21:04.367550 16879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:04.361901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36259 (local address 127.0.97.1:41017)
0928 05:21:04.362075 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 05:21:04.362079 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:21:04.362547 (+ 468us) server_negotiation.cc:407] Connection header received
0928 05:21:04.362788 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:21:04.362793 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:21:04.362871 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:21:04.362957 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:21:04.363511 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:04.364205 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:04.365920 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:04.366700 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:04.366818 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:04.366911 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 05:21:04.367179 (+ 268us) server_negotiation.cc:299] Negotiation successful
0928 05:21:04.367300 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 05:21:04.367614 16892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:04.361754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41017 (local address 127.0.97.2:36259)
0928 05:21:04.362448 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:04.362471 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 05:21:04.362612 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:21:04.362997 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:21:04.363010 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:21:04.363043 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:21:04.363373 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:04.363383 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:04.364335 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:04.364338 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:04.365774 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:04.365786 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:04.366902 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:04.366908 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:04.367040 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:04.367063 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 05:21:04.367142 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 05:21:04.367263 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":377,"threads_started":1}
I20260928 05:21:04.368059 16882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:04.362791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56173 (local address 127.0.97.3:36609)
0928 05:21:04.362989 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 05:21:04.362995 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:21:04.363221 (+ 226us) server_negotiation.cc:407] Connection header received
0928 05:21:04.363393 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:21:04.363397 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:21:04.363452 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:21:04.363519 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:21:04.364008 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:04.364697 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:04.366347 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:04.367110 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:04.367217 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:04.367311 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 05:21:04.367658 (+ 347us) server_negotiation.cc:299] Negotiation successful
0928 05:21:04.367781 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 05:21:04.368489 16893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:04.362580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36609 (local address 127.0.97.2:56173)
0928 05:21:04.363122 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:04.363136 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 05:21:04.363261 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:21:04.363553 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:21:04.363557 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:21:04.363579 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:21:04.363875 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:04.363883 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:04.364819 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:04.364828 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:04.366201 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:04.366213 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:04.367302 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:04.367307 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:04.367463 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:04.367485 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 05:21:04.368152 (+ 667us) client_negotiation.cc:240] Negotiation successful
0928 05:21:04.368274 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":411,"threads_started":1}
I20260928 05:21:04.368841 16559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "196ffebfe3f548ff8bf8674ab4773f25" candidate_uuid: "6f265296eba24246a0acd818360338ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3457ee2e0bfb450cbf6b432ce660d3b9" is_pre_election: true
I20260928 05:21:04.369354 16822 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "196ffebfe3f548ff8bf8674ab4773f25" candidate_uuid: "6f265296eba24246a0acd818360338ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4393522a68d64a33b6a85780a76e8e82" is_pre_election: true
I20260928 05:21:04.369431 16559 raft_consensus.cc:2454] T 196ffebfe3f548ff8bf8674ab4773f25 P 3457ee2e0bfb450cbf6b432ce660d3b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f265296eba24246a0acd818360338ef in term 0.
I20260928 05:21:04.369730 16822 raft_consensus.cc:2454] T 196ffebfe3f548ff8bf8674ab4773f25 P 4393522a68d64a33b6a85780a76e8e82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f265296eba24246a0acd818360338ef in term 0.
I20260928 05:21:04.370029 16626 leader_election.cc:302] T 196ffebfe3f548ff8bf8674ab4773f25 P 6f265296eba24246a0acd818360338ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3457ee2e0bfb450cbf6b432ce660d3b9, 6f265296eba24246a0acd818360338ef; no voters:
I20260928 05:21:04.370363 16888 raft_consensus.cc:2781] T 196ffebfe3f548ff8bf8674ab4773f25 P 6f265296eba24246a0acd818360338ef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 05:21:04.370512 16888 raft_consensus.cc:482] T 196ffebfe3f548ff8bf8674ab4773f25 P 6f265296eba24246a0acd818360338ef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 05:21:04.370616 16888 raft_consensus.cc:3037] T 196ffebfe3f548ff8bf8674ab4773f25 P 6f265296eba24246a0acd818360338ef [term 0 FOLLOWER]: Advancing to term 1
W20260928 05:21:04.370949 16868 tablet.cc:2369] T 196ffebfe3f548ff8bf8674ab4773f25 P 4393522a68d64a33b6a85780a76e8e82: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 05:21:04.372697 16888 raft_consensus.cc:504] T 196ffebfe3f548ff8bf8674ab4773f25 P 6f265296eba24246a0acd818360338ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f265296eba24246a0acd818360338ef" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44643 } } peers { permanent_uuid: "3457ee2e0bfb450cbf6b432ce660d3b9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41017 } } peers { permanent_uuid: "4393522a68d64a33b6a85780a76e8e82" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36609 } }
I20260928 05:21:04.373281 16888 leader_election.cc:288] T 196ffebfe3f548ff8bf8674ab4773f25 P 6f265296eba24246a0acd818360338ef [CANDIDATE]: Term 1 election: Requested vote from peers 3457ee2e0bfb450cbf6b432ce660d3b9 (127.0.97.1:41017), 4393522a68d64a33b6a85780a76e8e82 (127.0.97.3:36609)
I20260928 05:21:04.373649 16559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "196ffebfe3f548ff8bf8674ab4773f25" candidate_uuid: "6f265296eba24246a0acd818360338ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3457ee2e0bfb450cbf6b432ce660d3b9"
I20260928 05:21:04.373703 16822 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "196ffebfe3f548ff8bf8674ab4773f25" candidate_uuid: "6f265296eba24246a0acd818360338ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4393522a68d64a33b6a85780a76e8e82"
I20260928 05:21:04.373804 16559 raft_consensus.cc:3037] T 196ffebfe3f548ff8bf8674ab4773f25 P 3457ee2e0bfb450cbf6b432ce660d3b9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:21:04.373876 16822 raft_consensus.cc:3037] T 196ffebfe3f548ff8bf8674ab4773f25 P 4393522a68d64a33b6a85780a76e8e82 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:21:04.375862 16559 raft_consensus.cc:2454] T 196ffebfe3f548ff8bf8674ab4773f25 P 3457ee2e0bfb450cbf6b432ce660d3b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f265296eba24246a0acd818360338ef in term 1.
I20260928 05:21:04.375927 16822 raft_consensus.cc:2454] T 196ffebfe3f548ff8bf8674ab4773f25 P 4393522a68d64a33b6a85780a76e8e82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f265296eba24246a0acd818360338ef in term 1.
I20260928 05:21:04.376196 16626 leader_election.cc:302] T 196ffebfe3f548ff8bf8674ab4773f25 P 6f265296eba24246a0acd818360338ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3457ee2e0bfb450cbf6b432ce660d3b9, 6f265296eba24246a0acd818360338ef; no voters:
I20260928 05:21:04.376425 16888 raft_consensus.cc:2781] T 196ffebfe3f548ff8bf8674ab4773f25 P 6f265296eba24246a0acd818360338ef [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:21:04.376832 16888 raft_consensus.cc:686] T 196ffebfe3f548ff8bf8674ab4773f25 P 6f265296eba24246a0acd818360338ef [term 1 LEADER]: Becoming Leader. State: Replica: 6f265296eba24246a0acd818360338ef, State: Running, Role: LEADER
I20260928 05:21:04.377333 16888 consensus_queue.cc:237] T 196ffebfe3f548ff8bf8674ab4773f25 P 6f265296eba24246a0acd818360338ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f265296eba24246a0acd818360338ef" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44643 } } peers { permanent_uuid: "3457ee2e0bfb450cbf6b432ce660d3b9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41017 } } peers { permanent_uuid: "4393522a68d64a33b6a85780a76e8e82" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36609 } }
I20260928 05:21:04.381482 16419 catalog_manager.cc:5495] T 196ffebfe3f548ff8bf8674ab4773f25 P 6f265296eba24246a0acd818360338ef reported cstate change: term changed from 0 to 1, leader changed from <none> to 6f265296eba24246a0acd818360338ef (127.0.97.2). New cstate: current_term: 1 leader_uuid: "6f265296eba24246a0acd818360338ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f265296eba24246a0acd818360338ef" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3457ee2e0bfb450cbf6b432ce660d3b9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41017 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4393522a68d64a33b6a85780a76e8e82" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36609 } health_report { overall_health: UNKNOWN } } }
W20260928 05:21:04.431805 16737 tablet.cc:2369] T 196ffebfe3f548ff8bf8674ab4773f25 P 6f265296eba24246a0acd818360338ef: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 05:21:04.441989 16880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:04.436604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47252 (local address 127.0.97.2:44643)
0928 05:21:04.436760 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 05:21:04.436765 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:21:04.436781 (+ 16us) server_negotiation.cc:407] Connection header received
0928 05:21:04.436985 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:21:04.436989 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:21:04.437063 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:21:04.437178 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:21:04.437492 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:04.438170 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:04.439515 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:04.440057 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:04.440179 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:04.440719 (+ 540us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:21:04.441393 (+ 674us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:21:04.441537 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 05:21:04.441641 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 05:21:04.441718 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 05:21:04.450667 16822 raft_consensus.cc:1261] T 196ffebfe3f548ff8bf8674ab4773f25 P 4393522a68d64a33b6a85780a76e8e82 [term 1 FOLLOWER]: Refusing update from remote peer 6f265296eba24246a0acd818360338ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:21:04.450747 16559 raft_consensus.cc:1261] T 196ffebfe3f548ff8bf8674ab4773f25 P 3457ee2e0bfb450cbf6b432ce660d3b9 [term 1 FOLLOWER]: Refusing update from remote peer 6f265296eba24246a0acd818360338ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:21:04.451674 16894 consensus_queue.cc:1035] T 196ffebfe3f548ff8bf8674ab4773f25 P 6f265296eba24246a0acd818360338ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "3457ee2e0bfb450cbf6b432ce660d3b9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:21:04.452033 16888 consensus_queue.cc:1035] T 196ffebfe3f548ff8bf8674ab4773f25 P 6f265296eba24246a0acd818360338ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "4393522a68d64a33b6a85780a76e8e82" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:21:04.462251 16897 mvcc.cc:205] Tried to move back new op lower bound from 7334186452780834816 to 7334186452493373440. Current Snapshot: MvccSnapshot[applied={T|T < 7334186452780834816}]
I20260928 05:21:04.476402 16419 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40526:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:21:04.476680 16419 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40526:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:21:04.480129 16419 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fa08fd07ee1e42f8956f8a8464523f55: Sending DeleteTablet for 3 replicas of tablet 196ffebfe3f548ff8bf8674ab4773f25
I20260928 05:21:04.481294 16671 tablet_service.cc:1514] Processing DeleteTablet for tablet 196ffebfe3f548ff8bf8674ab4773f25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:21:04 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47244
I20260928 05:21:04.481294 16539 tablet_service.cc:1514] Processing DeleteTablet for tablet 196ffebfe3f548ff8bf8674ab4773f25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:21:04 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54982
I20260928 05:21:04.481495 16802 tablet_service.cc:1514] Processing DeleteTablet for tablet 196ffebfe3f548ff8bf8674ab4773f25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:21:04 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48536
I20260928 05:21:04.481624 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 16456
I20260928 05:21:04.499903 16879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:04.482367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54996 (local address 127.0.97.1:41017)
0928 05:21:04.482526 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 05:21:04.482531 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:21:04.482641 (+ 110us) server_negotiation.cc:407] Connection header received
0928 05:21:04.482806 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:21:04.482810 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:21:04.482892 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:21:04.482984 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:21:04.484665 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:04.485878 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:04.487802 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:04.488515 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:04.488626 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:04.492631 (+ 4005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:21:04.492650 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:21:04.492670 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:21:04.492715 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:21:04.496812 (+ 4097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:21:04.497702 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:21:04.497713 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:21:04.497718 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:21:04.497896 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:21:04.498270 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:21:04.498275 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:21:04.498279 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:21:04.498827 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:21:04.499108 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0928 05:21:04.499360 (+ 252us) server_negotiation.cc:299] Negotiation successful
0928 05:21:04.499557 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 05:21:04.500586 16599 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:21:04.506084 16907 tablet_replica.cc:330] T 196ffebfe3f548ff8bf8674ab4773f25 P 6f265296eba24246a0acd818360338ef: stopping tablet replica
I20260928 05:21:04.506592 16907 raft_consensus.cc:2229] T 196ffebfe3f548ff8bf8674ab4773f25 P 6f265296eba24246a0acd818360338ef [term 1 LEADER]: Raft consensus shutting down.
I20260928 05:21:04.507333 16907 raft_consensus.cc:2258] T 196ffebfe3f548ff8bf8674ab4773f25 P 6f265296eba24246a0acd818360338ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:21:04.509747 16907 ts_tablet_manager.cc:1905] T 196ffebfe3f548ff8bf8674ab4773f25 P 6f265296eba24246a0acd818360338ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:21:04.517297 16907 ts_tablet_manager.cc:1918] T 196ffebfe3f548ff8bf8674ab4773f25 P 6f265296eba24246a0acd818360338ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:21:04.517485 16907 log.cc:1196] T 196ffebfe3f548ff8bf8674ab4773f25 P 6f265296eba24246a0acd818360338ef: Deleting WAL directory at /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-1/wal/wals/196ffebfe3f548ff8bf8674ab4773f25
I20260928 05:21:04.517962 16909 tablet_replica.cc:330] T 196ffebfe3f548ff8bf8674ab4773f25 P 4393522a68d64a33b6a85780a76e8e82: stopping tablet replica
I20260928 05:21:04.518024 16907 ts_tablet_manager.cc:1939] T 196ffebfe3f548ff8bf8674ab4773f25 P 6f265296eba24246a0acd818360338ef: Deleting consensus metadata
I20260928 05:21:04.518445 16909 raft_consensus.cc:2229] T 196ffebfe3f548ff8bf8674ab4773f25 P 4393522a68d64a33b6a85780a76e8e82 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:21:04.518947 16909 raft_consensus.cc:2258] T 196ffebfe3f548ff8bf8674ab4773f25 P 4393522a68d64a33b6a85780a76e8e82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:21:04.519871 16403 catalog_manager.cc:4841] TS 6f265296eba24246a0acd818360338ef (127.0.97.2:44643): tablet 196ffebfe3f548ff8bf8674ab4773f25 (table test-table [id=8cc077ac2093419e80bf1bc4d5df2f3e]) successfully deleted
I20260928 05:21:04.521104 16909 ts_tablet_manager.cc:1905] T 196ffebfe3f548ff8bf8674ab4773f25 P 4393522a68d64a33b6a85780a76e8e82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:21:04.528235 16909 ts_tablet_manager.cc:1918] T 196ffebfe3f548ff8bf8674ab4773f25 P 4393522a68d64a33b6a85780a76e8e82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:21:04.528391 16909 log.cc:1196] T 196ffebfe3f548ff8bf8674ab4773f25 P 4393522a68d64a33b6a85780a76e8e82: Deleting WAL directory at /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-2/wal/wals/196ffebfe3f548ff8bf8674ab4773f25
I20260928 05:21:04.528883 16909 ts_tablet_manager.cc:1939] T 196ffebfe3f548ff8bf8674ab4773f25 P 4393522a68d64a33b6a85780a76e8e82: Deleting consensus metadata
I20260928 05:21:04.530651 16405 catalog_manager.cc:4841] TS 4393522a68d64a33b6a85780a76e8e82 (127.0.97.3:36609): tablet 196ffebfe3f548ff8bf8674ab4773f25 (table test-table [id=8cc077ac2093419e80bf1bc4d5df2f3e]) successfully deleted
I20260928 05:21:04.606775 16908 tablet_replica.cc:330] T 196ffebfe3f548ff8bf8674ab4773f25 P 3457ee2e0bfb450cbf6b432ce660d3b9: stopping tablet replica
I20260928 05:21:04.607200 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 16456
I20260928 05:21:04.607195 16908 raft_consensus.cc:2229] T 196ffebfe3f548ff8bf8674ab4773f25 P 3457ee2e0bfb450cbf6b432ce660d3b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:21:04.620946 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 16608
W20260928 05:21:04.621016 16405 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:41017 (error 108)
I20260928 05:21:04.623535 16877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:04.623057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41017 (local address 127.0.0.1:55000)
0928 05:21:04.623239 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:04.623351 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260928 05:21:04.623871 16405 catalog_manager.cc:4584] TS 3457ee2e0bfb450cbf6b432ce660d3b9 (127.0.97.1:41017): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 196ffebfe3f548ff8bf8674ab4773f25: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41017: connect: Connection refused (error 111)
I20260928 05:21:04.634865 16880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:04.621664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47254 (local address 127.0.97.2:44643)
0928 05:21:04.621846 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 05:21:04.621852 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:21:04.621927 (+ 75us) server_negotiation.cc:407] Connection header received
0928 05:21:04.622122 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:21:04.622128 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:21:04.622213 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:21:04.622304 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:21:04.624001 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:04.624822 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:04.626220 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:04.626692 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:04.626797 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:04.629272 (+ 2475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:21:04.629283 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:21:04.629301 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:21:04.629338 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:21:04.632109 (+ 2771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:21:04.632713 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:21:04.632722 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:21:04.632727 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:21:04.632922 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:21:04.633251 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:21:04.633257 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:21:04.633261 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:21:04.633614 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:21:04.633932 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0928 05:21:04.634175 (+ 243us) server_negotiation.cc:299] Negotiation successful
0928 05:21:04.634376 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 05:21:04.635628 16731 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:21:04.677529 16877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:04.677025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41017 (local address 127.0.0.1:55012)
0928 05:21:04.677230 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:04.677348 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260928 05:21:04.678516 16877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:04.678140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41017 (local address 127.0.0.1:55020)
0928 05:21:04.678276 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:04.678372 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260928 05:21:04.739076 16877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:04.738492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41017 (local address 127.0.0.1:55028)
0928 05:21:04.738730 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:04.738867 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260928 05:21:04.740155 16877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:04.739771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41017 (local address 127.0.0.1:55034)
0928 05:21:04.739923 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:04.740024 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260928 05:21:04.745030 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 16608
I20260928 05:21:04.759701 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 16739
I20260928 05:21:04.771572 16882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:04.760380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48542 (local address 127.0.97.3:36609)
0928 05:21:04.760536 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 05:21:04.760540 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:21:04.760559 (+ 19us) server_negotiation.cc:407] Connection header received
0928 05:21:04.760675 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:21:04.760678 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:21:04.760743 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:21:04.760828 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:21:04.762056 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:04.762727 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:04.763913 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:04.764293 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:04.764382 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:04.766731 (+ 2349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:21:04.766743 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:21:04.766759 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:21:04.766793 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:21:04.769412 (+ 2619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:21:04.770039 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:21:04.770045 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:21:04.770047 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:21:04.770173 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:21:04.770462 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:21:04.770465 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:21:04.770467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:21:04.770760 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:21:04.770969 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0928 05:21:04.771184 (+ 215us) server_negotiation.cc:299] Negotiation successful
0928 05:21:04.771327 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 05:21:04.772154 16862 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:21:04.820683 16877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:04.820160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41017 (local address 127.0.0.1:55048)
0928 05:21:04.820388 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:04.820497 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260928 05:21:04.821724 16877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:04.821310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41017 (local address 127.0.0.1:55064)
0928 05:21:04.821503 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:04.821598 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260928 05:21:04.870901 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 16739
I20260928 05:21:04.883903 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 16386
I20260928 05:21:04.884992 16449 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:21:04.997906 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 16386
I20260928 05:21:25.015318 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:32969
--webserver_interface=127.0.97.62
--webserver_port=36021
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:32969
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:21:25.324317 16915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:21:25.324600 16915 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:21:25.324717 16915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:21:25.331959 16915 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 05:21:25.332053 16915 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 05:21:25.332093 16915 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 05:21:25.332161 16915 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:21:25.332228 16915 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:21:25.332319 16915 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:21:25.332372 16915 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:21:25.332444 16915 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:21:25.342160 16915 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:32969
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:32969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=36021
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:21:25.343389 16915 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:21:25.344676 16915 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 05:21:25.356624 16915 server_base.cc:1034] running on GCE node
W20260928 05:21:25.356477 16921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:21:25.356529 16920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:21:25.356554 16923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:21:25.357733 16915 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:21:25.359100 16915 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:21:25.360319 16915 hybrid_clock.cc:648] HybridClock initialized: now 1790572885360255 us; error 40 us; skew 500 ppm
I20260928 05:21:25.360634 16915 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:21:25 dist-test-slave-whsn krb5kdc[15836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572885, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:21:25.363536 16915 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 05:21:25.364820 16915 webserver.cc:460] Webserver started at http://127.0.97.62:36021/ using document root <none> and password file <none>
I20260928 05:21:25.365489 16915 fs_manager.cc:359] Metadata directory not provided
I20260928 05:21:25.365592 16915 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:21:25.370672 16915 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260928 05:21:25.373260 16930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:21:25.374436 16915 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 05:21:25.374593 16915 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/master-0/wal
uuid: "fa08fd07ee1e42f8956f8a8464523f55"
format_stamp: "Formatted at 2026-09-28 05:20:40 on dist-test-slave-whsn"
server_key: "5283b56f3149cd0cafcc4d4b183d8b08"
server_key_iv: "0e3b1c8c08bb8235dec625a117da870f"
server_key_version: "encryptionkey@0"
I20260928 05:21:25.375447 16915 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:21:25.392923 16915 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:21:25.396502 16915 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:21:25.396759 16915 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:21:25.413060 16915 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:32969
I20260928 05:21:25.413092 16981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:32969 every 8 connection(s)
I20260928 05:21:25.414429 16915 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:21:25.417434 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 16915
I20260928 05:21:25.422672 16982 sys_catalog.cc:263] Verifying existing consensus state
I20260928 05:21:25.423070 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:41017
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:32969
--webserver_port=41847
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 05:21:25.426870 16982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa08fd07ee1e42f8956f8a8464523f55: Bootstrap starting.
I20260928 05:21:25.446764 16982 log.cc:824] T 00000000000000000000000000000000 P fa08fd07ee1e42f8956f8a8464523f55: Log is configured to *not* fsync() on all Append() calls
I20260928 05:21:25.461858 16982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa08fd07ee1e42f8956f8a8464523f55: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 05:21:25.462299 16982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa08fd07ee1e42f8956f8a8464523f55: Bootstrap complete.
I20260928 05:21:25.470748 16982 raft_consensus.cc:348] T 00000000000000000000000000000000 P fa08fd07ee1e42f8956f8a8464523f55 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa08fd07ee1e42f8956f8a8464523f55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32969 } }
I20260928 05:21:25.471542 16982 raft_consensus.cc:729] T 00000000000000000000000000000000 P fa08fd07ee1e42f8956f8a8464523f55 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa08fd07ee1e42f8956f8a8464523f55, State: Initialized, Role: FOLLOWER
I20260928 05:21:25.472149 16982 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa08fd07ee1e42f8956f8a8464523f55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa08fd07ee1e42f8956f8a8464523f55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32969 } }
I20260928 05:21:25.472371 16982 raft_consensus.cc:388] T 00000000000000000000000000000000 P fa08fd07ee1e42f8956f8a8464523f55 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:21:25.472478 16982 raft_consensus.cc:482] T 00000000000000000000000000000000 P fa08fd07ee1e42f8956f8a8464523f55 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:21:25.472630 16982 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa08fd07ee1e42f8956f8a8464523f55 [term 2 FOLLOWER]: Advancing to term 3
I20260928 05:21:25.476073 16982 raft_consensus.cc:504] T 00000000000000000000000000000000 P fa08fd07ee1e42f8956f8a8464523f55 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa08fd07ee1e42f8956f8a8464523f55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32969 } }
I20260928 05:21:25.476527 16982 leader_election.cc:302] T 00000000000000000000000000000000 P fa08fd07ee1e42f8956f8a8464523f55 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fa08fd07ee1e42f8956f8a8464523f55; no voters:
I20260928 05:21:25.477116 16982 leader_election.cc:288] T 00000000000000000000000000000000 P fa08fd07ee1e42f8956f8a8464523f55 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 05:21:25.477241 16986 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fa08fd07ee1e42f8956f8a8464523f55 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 05:21:25.478313 16986 raft_consensus.cc:686] T 00000000000000000000000000000000 P fa08fd07ee1e42f8956f8a8464523f55 [term 3 LEADER]: Becoming Leader. State: Replica: fa08fd07ee1e42f8956f8a8464523f55, State: Running, Role: LEADER
I20260928 05:21:25.478905 16986 consensus_queue.cc:237] T 00000000000000000000000000000000 P fa08fd07ee1e42f8956f8a8464523f55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa08fd07ee1e42f8956f8a8464523f55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32969 } }
I20260928 05:21:25.479197 16982 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa08fd07ee1e42f8956f8a8464523f55 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:21:25.481251 16988 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa08fd07ee1e42f8956f8a8464523f55 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa08fd07ee1e42f8956f8a8464523f55. Latest consensus state: current_term: 3 leader_uuid: "fa08fd07ee1e42f8956f8a8464523f55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa08fd07ee1e42f8956f8a8464523f55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32969 } } }
I20260928 05:21:25.481611 16988 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa08fd07ee1e42f8956f8a8464523f55 [sys.catalog]: This master's current role is: LEADER
I20260928 05:21:25.482110 16987 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa08fd07ee1e42f8956f8a8464523f55 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "fa08fd07ee1e42f8956f8a8464523f55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa08fd07ee1e42f8956f8a8464523f55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32969 } } }
I20260928 05:21:25.482375 16987 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa08fd07ee1e42f8956f8a8464523f55 [sys.catalog]: This master's current role is: LEADER
I20260928 05:21:25.491817 16994 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:21:25.499969 16994 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:21:25.501448 16994 catalog_manager.cc:1259] Loaded cluster ID: 267ca691bdf34212ba7e1a115574519b
I20260928 05:21:25.501596 16994 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:21:25.505623 16994 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:21:25.507269 16994 catalog_manager.cc:5879] T 00000000000000000000000000000000 P fa08fd07ee1e42f8956f8a8464523f55: Loaded TSK: 0
I20260928 05:21:25.507441 16994 catalog_manager.cc:5879] T 00000000000000000000000000000000 P fa08fd07ee1e42f8956f8a8464523f55: Loaded TSK: 1
F20260928 05:21:25.508538 16994 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f50fb5ef9ad google::LogMessage::Fail() at ??:0
@ 0x7f50fb5f288a google::LogMessage::SendToLog() at ??:0
@ 0x7f50fb5ef4de google::LogMessage::Flush() at ??:0
@ 0x7f50fb5f0a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f50ff3537d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f50ff34273e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f511101e1c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f511101d783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f51110a92a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f51110a90c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f51110f35e9 std::function<>::operator()() at ??:0
@ 0x7f5111021cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f511101fc36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f51110a7610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f51110a7439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f50fe23b345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f50fe23e91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f50fe23e799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f50fe219eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f51014f36db start_thread at ??:0
@ 0x7f50fa48961f clone at ??:0
W20260928 05:21:25.766305 16984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:21:25.766564 16984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:21:25.766695 16984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:21:25.773739 16984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:21:25.773840 16984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:21:25.773993 16984 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 05:21:25.783215 16984 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:41017
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=41847
--tserver_master_addrs=127.0.97.62:32969
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:21:25.784356 16984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:21:25.785615 16984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:21:25.797299 17030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:21:25.797348 17029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:21:25.797878 17032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:21:25.798393 16984 server_base.cc:1034] running on GCE node
I20260928 05:21:25.798938 16984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:21:25.799814 16984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:21:25.800993 16984 hybrid_clock.cc:648] HybridClock initialized: now 1790572885800940 us; error 46 us; skew 500 ppm
I20260928 05:21:25.801270 16984 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:21:25 dist-test-slave-whsn krb5kdc[15836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572885, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:21:25.807994 16984 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260928 05:21:25.809269 16984 webserver.cc:460] Webserver started at http://127.0.97.1:41847/ using document root <none> and password file <none>
I20260928 05:21:25.809867 16984 fs_manager.cc:359] Metadata directory not provided
I20260928 05:21:25.809984 16984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:21:25.814812 16984 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260928 05:21:25.818647 17039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:21:25.819778 16984 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 05:21:25.819908 16984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-0/wal
uuid: "3457ee2e0bfb450cbf6b432ce660d3b9"
format_stamp: "Formatted at 2026-09-28 05:20:40 on dist-test-slave-whsn"
server_key: "caf76cff06d56cf1c8aaa9db349c62a9"
server_key_iv: "a44742f50c5b0d7138129bffb82fa618"
server_key_version: "encryptionkey@0"
I20260928 05:21:25.820767 16984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:21:25.841300 16984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:21:25.844561 16984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:21:25.844805 16984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:21:25.845654 16984 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:21:25.847214 17046 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260928 05:21:25.852262 16984 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 05:21:25.852366 16984 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.001s
I20260928 05:21:25.852466 16984 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 05:21:25.855407 16984 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 05:21:25.855487 16984 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.000s sys 0.001s
I20260928 05:21:25.855782 17046 tablet_bootstrap.cc:492] T 196ffebfe3f548ff8bf8674ab4773f25 P 3457ee2e0bfb450cbf6b432ce660d3b9: Bootstrap starting.
I20260928 05:21:25.880743 17046 log.cc:824] T 196ffebfe3f548ff8bf8674ab4773f25 P 3457ee2e0bfb450cbf6b432ce660d3b9: Log is configured to *not* fsync() on all Append() calls
I20260928 05:21:25.888216 16984 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41017
I20260928 05:21:25.888239 17153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41017 every 8 connection(s)
I20260928 05:21:25.889739 16984 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-0/data/info.pb
I20260928 05:21:25.893708 17046 tablet_bootstrap.cc:492] T 196ffebfe3f548ff8bf8674ab4773f25 P 3457ee2e0bfb450cbf6b432ce660d3b9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 05:21:25.894173 17046 tablet_bootstrap.cc:492] T 196ffebfe3f548ff8bf8674ab4773f25 P 3457ee2e0bfb450cbf6b432ce660d3b9: Bootstrap complete.
I20260928 05:21:25.894220 17156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:25.892232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32969 (local address 127.0.97.1:49147)
0928 05:21:25.893075 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:25.893304 (+ 229us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":339,"threads_started":1}
W20260928 05:21:25.894872 17154 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:32969 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:32969: Client connection negotiation failed: client connection to 127.0.97.62:32969: connect: Connection refused (error 111)
I20260928 05:21:25.895231 17046 ts_tablet_manager.cc:1397] T 196ffebfe3f548ff8bf8674ab4773f25 P 3457ee2e0bfb450cbf6b432ce660d3b9: Time spent bootstrapping tablet: real 0.040s user 0.029s sys 0.008s
I20260928 05:21:25.895704 17156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:25.895280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32969 (local address 127.0.97.1:59333)
0928 05:21:25.895498 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:25.895582 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260928 05:21:25.895771 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 16984
I20260928 05:21:25.896445 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:44643
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:32969
--webserver_port=40821
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 05:21:25.896742 17156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:25.896332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32969 (local address 127.0.97.1:57901)
0928 05:21:25.896481 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:25.896584 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260928 05:21:25.897001 17154 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 05:21:25.901434 17046 raft_consensus.cc:348] T 196ffebfe3f548ff8bf8674ab4773f25 P 3457ee2e0bfb450cbf6b432ce660d3b9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f265296eba24246a0acd818360338ef" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44643 } } peers { permanent_uuid: "3457ee2e0bfb450cbf6b432ce660d3b9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41017 } } peers { permanent_uuid: "4393522a68d64a33b6a85780a76e8e82" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36609 } }
I20260928 05:21:25.902232 17046 raft_consensus.cc:729] T 196ffebfe3f548ff8bf8674ab4773f25 P 3457ee2e0bfb450cbf6b432ce660d3b9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3457ee2e0bfb450cbf6b432ce660d3b9, State: Initialized, Role: FOLLOWER
I20260928 05:21:25.902766 17046 consensus_queue.cc:260] T 196ffebfe3f548ff8bf8674ab4773f25 P 3457ee2e0bfb450cbf6b432ce660d3b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f265296eba24246a0acd818360338ef" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44643 } } peers { permanent_uuid: "3457ee2e0bfb450cbf6b432ce660d3b9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41017 } } peers { permanent_uuid: "4393522a68d64a33b6a85780a76e8e82" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36609 } }
I20260928 05:21:25.904071 17046 ts_tablet_manager.cc:1428] T 196ffebfe3f548ff8bf8674ab4773f25 P 3457ee2e0bfb450cbf6b432ce660d3b9: Time spent starting tablet: real 0.009s user 0.010s sys 0.000s
I20260928 05:21:25.904220 17156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:25.903837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32969 (local address 127.0.97.1:40087)
0928 05:21:25.903983 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:25.904055 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260928 05:21:25.905090 17156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:25.904722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32969 (local address 127.0.97.1:39379)
0928 05:21:25.904882 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:25.904963 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260928 05:21:26.209120 17157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:21:26.209404 17157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:21:26.209512 17157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:21:26.216722 17157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:21:26.216835 17157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:21:26.217028 17157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 05:21:26.226672 17157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:44643
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=40821
--tserver_master_addrs=127.0.97.62:32969
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:21:26.227836 17157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:21:26.229131 17157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:21:26.241791 17166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:21:26.241830 17163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:21:26.241871 17164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:21:26.243031 17157 server_base.cc:1034] running on GCE node
I20260928 05:21:26.243577 17157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:21:26.244431 17157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:21:26.245617 17157 hybrid_clock.cc:648] HybridClock initialized: now 1790572886245569 us; error 38 us; skew 500 ppm
I20260928 05:21:26.245908 17157 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:21:26 dist-test-slave-whsn krb5kdc[15836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572886, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:21:26.248744 17157 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260928 05:21:26.250095 17157 webserver.cc:460] Webserver started at http://127.0.97.2:40821/ using document root <none> and password file <none>
I20260928 05:21:26.250725 17157 fs_manager.cc:359] Metadata directory not provided
I20260928 05:21:26.250834 17157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:21:26.255877 17157 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 05:21:26.258422 17173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:21:26.259658 17157 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 05:21:26.259804 17157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-1/wal
uuid: "6f265296eba24246a0acd818360338ef"
format_stamp: "Formatted at 2026-09-28 05:20:41 on dist-test-slave-whsn"
server_key: "ce39ec02e78df8127061b816f18938e2"
server_key_iv: "7c8ac7bd314af6b6bb1a6bc983843871"
server_key_version: "encryptionkey@0"
I20260928 05:21:26.260716 17157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:21:26.281698 17157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:21:26.285297 17157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:21:26.285557 17157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:21:26.286473 17157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:21:26.287810 17157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:21:26.287909 17157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:21:26.288008 17157 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:21:26.288079 17157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:21:26.316072 17157 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44643
I20260928 05:21:26.316145 17285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44643 every 8 connection(s)
I20260928 05:21:26.317397 17157 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-1/data/info.pb
I20260928 05:21:26.321827 17288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:26.320121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32969 (local address 127.0.97.2:53317)
0928 05:21:26.320773 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:26.321010 (+ 237us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":242,"threads_started":1}
W20260928 05:21:26.322458 17286 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:32969 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:32969: Client connection negotiation failed: client connection to 127.0.97.62:32969: connect: Connection refused (error 111)
I20260928 05:21:26.323138 17288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:26.322865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32969 (local address 127.0.97.2:53277)
0928 05:21:26.322982 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:26.323043 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260928 05:21:26.323904 17288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:26.323615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32969 (local address 127.0.97.2:41017)
0928 05:21:26.323723 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:26.323789 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260928 05:21:26.324102 17286 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 05:21:26.326856 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 17157
I20260928 05:21:26.327401 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:36609
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:32969
--webserver_port=44341
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:21:26.631090 17289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:21:26.631338 17289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:21:26.631465 17289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:21:26.638351 17289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:21:26.638453 17289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:21:26.638602 17289 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 05:21:26.647583 17289 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:36609
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=44341
--tserver_master_addrs=127.0.97.62:32969
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:21:26.648708 17289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:21:26.649921 17289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:21:26.661913 17295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:21:26.661962 17297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:21:26.661983 17294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:21:26.662663 17289 server_base.cc:1034] running on GCE node
I20260928 05:21:26.663177 17289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:21:26.664057 17289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:21:26.665243 17289 hybrid_clock.cc:648] HybridClock initialized: now 1790572886665179 us; error 56 us; skew 500 ppm
I20260928 05:21:26.665532 17289 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:21:26 dist-test-slave-whsn krb5kdc[15836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572886, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:21:26.668227 17289 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260928 05:21:26.669497 17289 webserver.cc:460] Webserver started at http://127.0.97.3:44341/ using document root <none> and password file <none>
I20260928 05:21:26.670080 17289 fs_manager.cc:359] Metadata directory not provided
I20260928 05:21:26.670183 17289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:21:26.674990 17289 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 05:21:26.677364 17304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:21:26.678563 17289 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 05:21:26.678689 17289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-2/wal
uuid: "4393522a68d64a33b6a85780a76e8e82"
format_stamp: "Formatted at 2026-09-28 05:20:41 on dist-test-slave-whsn"
server_key: "64e37ebd3b46409260d3eda232db36a9"
server_key_iv: "1a7bae69e865db0cf3ff452037c4e152"
server_key_version: "encryptionkey@0"
I20260928 05:21:26.679553 17289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:21:26.708997 17289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:21:26.712443 17289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:21:26.712675 17289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:21:26.713562 17289 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:21:26.714689 17289 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:21:26.714792 17289 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:21:26.714881 17289 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:21:26.714938 17289 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:21:26.742573 17289 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36609
I20260928 05:21:26.742640 17416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36609 every 8 connection(s)
I20260928 05:21:26.743808 17289 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572622576160-388-0/minicluster-data/ts-2/data/info.pb
I20260928 05:21:26.745419 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 17289
I20260928 05:21:26.748348 17420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:26.746466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32969 (local address 127.0.97.3:41543)
0928 05:21:26.747308 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:26.747538 (+ 230us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":300,"threads_started":1}
W20260928 05:21:26.748955 17417 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:32969 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:32969: Client connection negotiation failed: client connection to 127.0.97.62:32969: connect: Connection refused (error 111)
I20260928 05:21:26.749642 17420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:26.749358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32969 (local address 127.0.97.3:56987)
0928 05:21:26.749487 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:26.749545 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 05:21:26.750329 17420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:26.750046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32969 (local address 127.0.97.3:33581)
0928 05:21:26.750167 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:26.750232 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260928 05:21:26.750521 17417 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 05:21:26.906461 17421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:26.905793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32969 (local address 127.0.97.1:53163)
0928 05:21:26.906188 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:26.906276 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":187,"threads_started":1}
I20260928 05:21:27.065131 17422 raft_consensus.cc:482] T 196ffebfe3f548ff8bf8674ab4773f25 P 3457ee2e0bfb450cbf6b432ce660d3b9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:21:27.065418 17422 raft_consensus.cc:504] T 196ffebfe3f548ff8bf8674ab4773f25 P 3457ee2e0bfb450cbf6b432ce660d3b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f265296eba24246a0acd818360338ef" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44643 } } peers { permanent_uuid: "3457ee2e0bfb450cbf6b432ce660d3b9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41017 } } peers { permanent_uuid: "4393522a68d64a33b6a85780a76e8e82" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36609 } }
I20260928 05:21:27.067191 17422 leader_election.cc:288] T 196ffebfe3f548ff8bf8674ab4773f25 P 3457ee2e0bfb450cbf6b432ce660d3b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6f265296eba24246a0acd818360338ef (127.0.97.2:44643), 4393522a68d64a33b6a85780a76e8e82 (127.0.97.3:36609)
Sep 28 05:21:27 dist-test-slave-whsn krb5kdc[15836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572885, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
Sep 28 05:21:27 dist-test-slave-whsn krb5kdc[15836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572885, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260928 05:21:27.088340 17421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:27.067307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44643 (local address 127.0.97.1:46549)
0928 05:21:27.067485 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:27.067523 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 05:21:27.068058 (+ 535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:21:27.071558 (+ 3500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:21:27.071567 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:21:27.072360 (+ 793us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:21:27.073143 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:27.073163 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:27.074653 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:27.074656 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:27.075906 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:27.075917 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:27.076928 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:27.076936 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:27.077171 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:27.077907 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:21:27.077924 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:21:27.080250 (+ 2326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:21:27.084302 (+ 4052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:21:27.084310 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:21:27.084324 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:21:27.085052 (+ 728us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:21:27.085488 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:21:27.085493 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:21:27.085496 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:21:27.085752 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:21:27.086659 (+ 907us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:21:27.086667 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:21:27.087095 (+ 428us) client_negotiation.cc:769] Sending connection context
0928 05:21:27.087385 (+ 290us) client_negotiation.cc:240] Negotiation successful
0928 05:21:27.087840 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":161,"spinlock_wait_cycles":41728}
I20260928 05:21:27.088349 17424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:27.067542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36609 (local address 127.0.97.1:48509)
0928 05:21:27.068798 (+ 1256us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:27.068814 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 05:21:27.068949 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:21:27.071555 (+ 2606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:21:27.071566 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:21:27.072361 (+ 795us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:21:27.073146 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:27.073165 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:27.074646 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:27.074650 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:27.075945 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:27.075957 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:27.076888 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:27.076895 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:27.077166 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:27.077851 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:21:27.077875 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:21:27.080655 (+ 2780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:21:27.084304 (+ 3649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:21:27.084310 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:21:27.084324 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:21:27.085052 (+ 728us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:21:27.085496 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:21:27.085500 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:21:27.085503 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:21:27.085752 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:21:27.086659 (+ 907us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:21:27.086667 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:21:27.087096 (+ 429us) client_negotiation.cc:769] Sending connection context
0928 05:21:27.087385 (+ 289us) client_negotiation.cc:240] Negotiation successful
0928 05:21:27.087958 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1134,"mutex_wait_us":402,"spinlock_wait_cycles":50688,"thread_start_us":950,"threads_started":1}
I20260928 05:21:27.088430 17423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:27.067562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46549 (local address 127.0.97.2:44643)
0928 05:21:27.069338 (+ 1776us) server_negotiation.cc:206] Beginning negotiation
0928 05:21:27.069349 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 05:21:27.069385 (+ 36us) server_negotiation.cc:407] Connection header received
0928 05:21:27.070740 (+ 1355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:21:27.070748 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:21:27.071107 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:21:27.071379 (+ 272us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:21:27.073324 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:27.074477 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:27.076040 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:27.076679 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:27.076835 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:27.080515 (+ 3680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:21:27.080523 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:21:27.080535 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:21:27.080573 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:21:27.084168 (+ 3595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:21:27.085171 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:21:27.085180 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:21:27.085188 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:21:27.085379 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:21:27.085893 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:21:27.085898 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:21:27.085902 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:21:27.086512 (+ 610us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:21:27.086707 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0928 05:21:27.087637 (+ 930us) server_negotiation.cc:299] Negotiation successful
0928 05:21:27.088039 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1640,"thread_start_us":1552,"threads_started":1}
I20260928 05:21:27.089067 17425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:27.067959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48509 (local address 127.0.97.3:36609)
0928 05:21:27.069313 (+ 1354us) server_negotiation.cc:206] Beginning negotiation
0928 05:21:27.069324 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 05:21:27.069359 (+ 35us) server_negotiation.cc:407] Connection header received
0928 05:21:27.070822 (+ 1463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:21:27.070830 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:21:27.071165 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:21:27.071460 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:21:27.073323 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:27.074475 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:27.076090 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:27.076675 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:27.076835 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:27.080926 (+ 4091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:21:27.080936 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:21:27.080953 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:21:27.080996 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:21:27.084168 (+ 3172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:21:27.085173 (+ 1005us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:21:27.085181 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:21:27.085188 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:21:27.085388 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:21:27.085893 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:21:27.085898 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:21:27.085902 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:21:27.086512 (+ 610us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:21:27.086707 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0928 05:21:27.087641 (+ 934us) server_negotiation.cc:299] Negotiation successful
0928 05:21:27.088026 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1183,"thread_start_us":907,"threads_started":1}
W20260928 05:21:27.091702 17040 leader_election.cc:341] T 196ffebfe3f548ff8bf8674ab4773f25 P 3457ee2e0bfb450cbf6b432ce660d3b9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6f265296eba24246a0acd818360338ef (127.0.97.2:44643): Not found: Tablet not found: 196ffebfe3f548ff8bf8674ab4773f25
W20260928 05:21:27.092127 17042 leader_election.cc:341] T 196ffebfe3f548ff8bf8674ab4773f25 P 3457ee2e0bfb450cbf6b432ce660d3b9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4393522a68d64a33b6a85780a76e8e82 (127.0.97.3:36609): Not found: Tablet not found: 196ffebfe3f548ff8bf8674ab4773f25
I20260928 05:21:27.092283 17042 leader_election.cc:302] T 196ffebfe3f548ff8bf8674ab4773f25 P 3457ee2e0bfb450cbf6b432ce660d3b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3457ee2e0bfb450cbf6b432ce660d3b9; no voters: 4393522a68d64a33b6a85780a76e8e82, 6f265296eba24246a0acd818360338ef
I20260928 05:21:27.092623 17422 raft_consensus.cc:2726] T 196ffebfe3f548ff8bf8674ab4773f25 P 3457ee2e0bfb450cbf6b432ce660d3b9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 05:21:27.325510 17426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:27.324663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32969 (local address 127.0.97.2:49841)
0928 05:21:27.325151 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:27.325279 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":216,"threads_started":1}
I20260928 05:21:27.751721 17427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:27.751047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32969 (local address 127.0.97.3:34415)
0928 05:21:27.751427 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:27.751536 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":177,"threads_started":1}
I20260928 05:21:27.908056 17428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:27.907345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32969 (local address 127.0.97.1:39537)
0928 05:21:27.907801 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:27.907898 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":221,"threads_started":1}
I20260928 05:21:28.326851 17429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:28.326217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32969 (local address 127.0.97.2:42029)
0928 05:21:28.326639 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:28.326711 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":229,"threads_started":1}
I20260928 05:21:28.753136 17430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:28.752453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32969 (local address 127.0.97.3:46255)
0928 05:21:28.752881 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:28.752953 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":208,"threads_started":1}
I20260928 05:21:28.909575 17431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:28.908836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32969 (local address 127.0.97.1:47743)
0928 05:21:28.909330 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:28.909402 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":211,"threads_started":1}
I20260928 05:21:29.021785 17432 raft_consensus.cc:482] T 196ffebfe3f548ff8bf8674ab4773f25 P 3457ee2e0bfb450cbf6b432ce660d3b9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:21:29.021958 17432 raft_consensus.cc:504] T 196ffebfe3f548ff8bf8674ab4773f25 P 3457ee2e0bfb450cbf6b432ce660d3b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f265296eba24246a0acd818360338ef" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44643 } } peers { permanent_uuid: "3457ee2e0bfb450cbf6b432ce660d3b9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41017 } } peers { permanent_uuid: "4393522a68d64a33b6a85780a76e8e82" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36609 } }
I20260928 05:21:29.022540 17432 leader_election.cc:288] T 196ffebfe3f548ff8bf8674ab4773f25 P 3457ee2e0bfb450cbf6b432ce660d3b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6f265296eba24246a0acd818360338ef (127.0.97.2:44643), 4393522a68d64a33b6a85780a76e8e82 (127.0.97.3:36609)
W20260928 05:21:29.023470 17042 leader_election.cc:341] T 196ffebfe3f548ff8bf8674ab4773f25 P 3457ee2e0bfb450cbf6b432ce660d3b9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4393522a68d64a33b6a85780a76e8e82 (127.0.97.3:36609): Not found: Tablet not found: 196ffebfe3f548ff8bf8674ab4773f25
W20260928 05:21:29.023625 17040 leader_election.cc:341] T 196ffebfe3f548ff8bf8674ab4773f25 P 3457ee2e0bfb450cbf6b432ce660d3b9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6f265296eba24246a0acd818360338ef (127.0.97.2:44643): Not found: Tablet not found: 196ffebfe3f548ff8bf8674ab4773f25
I20260928 05:21:29.023772 17040 leader_election.cc:302] T 196ffebfe3f548ff8bf8674ab4773f25 P 3457ee2e0bfb450cbf6b432ce660d3b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3457ee2e0bfb450cbf6b432ce660d3b9; no voters: 4393522a68d64a33b6a85780a76e8e82, 6f265296eba24246a0acd818360338ef
I20260928 05:21:29.024011 17432 raft_consensus.cc:2726] T 196ffebfe3f548ff8bf8674ab4773f25 P 3457ee2e0bfb450cbf6b432ce660d3b9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 05:21:29.328250 17433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:29.327606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32969 (local address 127.0.97.2:43203)
0928 05:21:29.328014 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:29.328101 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":215,"threads_started":1}
I20260928 05:21:29.754441 17434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:29.753846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32969 (local address 127.0.97.3:41723)
0928 05:21:29.754212 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:29.754286 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":194,"threads_started":1}
I20260928 05:21:29.911044 17435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:29.910353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32969 (local address 127.0.97.1:49509)
0928 05:21:29.910776 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:29.910860 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":221,"threads_started":1}
I20260928 05:21:30.329679 17436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:30.329029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32969 (local address 127.0.97.2:37025)
0928 05:21:30.329429 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:30.329503 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":198,"threads_started":1}
I20260928 05:21:30.756134 17437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:30.755279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32969 (local address 127.0.97.3:49203)
0928 05:21:30.755770 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:30.755882 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":248,"threads_started":1}
I20260928 05:21:30.813259 17438 raft_consensus.cc:482] T 196ffebfe3f548ff8bf8674ab4773f25 P 3457ee2e0bfb450cbf6b432ce660d3b9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:21:30.813457 17438 raft_consensus.cc:504] T 196ffebfe3f548ff8bf8674ab4773f25 P 3457ee2e0bfb450cbf6b432ce660d3b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f265296eba24246a0acd818360338ef" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44643 } } peers { permanent_uuid: "3457ee2e0bfb450cbf6b432ce660d3b9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41017 } } peers { permanent_uuid: "4393522a68d64a33b6a85780a76e8e82" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36609 } }
I20260928 05:21:30.814088 17438 leader_election.cc:288] T 196ffebfe3f548ff8bf8674ab4773f25 P 3457ee2e0bfb450cbf6b432ce660d3b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6f265296eba24246a0acd818360338ef (127.0.97.2:44643), 4393522a68d64a33b6a85780a76e8e82 (127.0.97.3:36609)
W20260928 05:21:30.814960 17042 leader_election.cc:341] T 196ffebfe3f548ff8bf8674ab4773f25 P 3457ee2e0bfb450cbf6b432ce660d3b9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4393522a68d64a33b6a85780a76e8e82 (127.0.97.3:36609): Not found: Tablet not found: 196ffebfe3f548ff8bf8674ab4773f25
W20260928 05:21:30.815100 17040 leader_election.cc:341] T 196ffebfe3f548ff8bf8674ab4773f25 P 3457ee2e0bfb450cbf6b432ce660d3b9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6f265296eba24246a0acd818360338ef (127.0.97.2:44643): Not found: Tablet not found: 196ffebfe3f548ff8bf8674ab4773f25
I20260928 05:21:30.815218 17040 leader_election.cc:302] T 196ffebfe3f548ff8bf8674ab4773f25 P 3457ee2e0bfb450cbf6b432ce660d3b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3457ee2e0bfb450cbf6b432ce660d3b9; no voters: 4393522a68d64a33b6a85780a76e8e82, 6f265296eba24246a0acd818360338ef
I20260928 05:21:30.815429 17438 raft_consensus.cc:2726] T 196ffebfe3f548ff8bf8674ab4773f25 P 3457ee2e0bfb450cbf6b432ce660d3b9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 05:21:30.912727 17439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:30.911978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32969 (local address 127.0.97.1:59809)
0928 05:21:30.912440 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:30.912546 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":227,"threads_started":1}
I20260928 05:21:31.331185 17440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:31.330509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32969 (local address 127.0.97.2:53493)
0928 05:21:31.330953 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:31.331028 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":241,"threads_started":1}
I20260928 05:21:31.757753 17441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:31.757047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32969 (local address 127.0.97.3:57955)
0928 05:21:31.757506 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:31.757590 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":226,"threads_started":1}
I20260928 05:21:31.914258 17442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:31.913544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32969 (local address 127.0.97.1:45307)
0928 05:21:31.914000 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:31.914093 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":232,"threads_started":1}
I20260928 05:21:32.332690 17443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:32.331991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32969 (local address 127.0.97.2:33647)
0928 05:21:32.332447 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:32.332522 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":244,"threads_started":1}
I20260928 05:21:32.542520 17444 raft_consensus.cc:482] T 196ffebfe3f548ff8bf8674ab4773f25 P 3457ee2e0bfb450cbf6b432ce660d3b9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:21:32.542687 17444 raft_consensus.cc:504] T 196ffebfe3f548ff8bf8674ab4773f25 P 3457ee2e0bfb450cbf6b432ce660d3b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f265296eba24246a0acd818360338ef" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44643 } } peers { permanent_uuid: "3457ee2e0bfb450cbf6b432ce660d3b9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41017 } } peers { permanent_uuid: "4393522a68d64a33b6a85780a76e8e82" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36609 } }
I20260928 05:21:32.543329 17444 leader_election.cc:288] T 196ffebfe3f548ff8bf8674ab4773f25 P 3457ee2e0bfb450cbf6b432ce660d3b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6f265296eba24246a0acd818360338ef (127.0.97.2:44643), 4393522a68d64a33b6a85780a76e8e82 (127.0.97.3:36609)
W20260928 05:21:32.544217 17040 leader_election.cc:341] T 196ffebfe3f548ff8bf8674ab4773f25 P 3457ee2e0bfb450cbf6b432ce660d3b9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6f265296eba24246a0acd818360338ef (127.0.97.2:44643): Not found: Tablet not found: 196ffebfe3f548ff8bf8674ab4773f25
W20260928 05:21:32.544380 17042 leader_election.cc:341] T 196ffebfe3f548ff8bf8674ab4773f25 P 3457ee2e0bfb450cbf6b432ce660d3b9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4393522a68d64a33b6a85780a76e8e82 (127.0.97.3:36609): Not found: Tablet not found: 196ffebfe3f548ff8bf8674ab4773f25
I20260928 05:21:32.544513 17042 leader_election.cc:302] T 196ffebfe3f548ff8bf8674ab4773f25 P 3457ee2e0bfb450cbf6b432ce660d3b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3457ee2e0bfb450cbf6b432ce660d3b9; no voters: 4393522a68d64a33b6a85780a76e8e82, 6f265296eba24246a0acd818360338ef
I20260928 05:21:32.544746 17444 raft_consensus.cc:2726] T 196ffebfe3f548ff8bf8674ab4773f25 P 3457ee2e0bfb450cbf6b432ce660d3b9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 05:21:32.759186 17445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:32.758536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32969 (local address 127.0.97.3:37239)
0928 05:21:32.758950 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:32.759024 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":216,"threads_started":1}
I20260928 05:21:32.915795 17446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:32.915086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32969 (local address 127.0.97.1:38943)
0928 05:21:32.915512 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:32.915597 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":233,"threads_started":1}
I20260928 05:21:33.334183 17447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:33.333449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32969 (local address 127.0.97.2:33491)
0928 05:21:33.333923 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:33.334004 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":219,"threads_started":1}
I20260928 05:21:33.761199 17448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:33.760252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32969 (local address 127.0.97.3:35565)
0928 05:21:33.760811 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:33.760971 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":256,"threads_started":1}
I20260928 05:21:33.917557 17449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:33.916773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32969 (local address 127.0.97.1:46303)
0928 05:21:33.917247 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:33.917367 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":247,"threads_started":1}
I20260928 05:21:34.335628 17450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:34.334987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32969 (local address 127.0.97.2:50899)
0928 05:21:34.335368 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:34.335455 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":190,"threads_started":1}
I20260928 05:21:34.762671 17451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:34.762036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32969 (local address 127.0.97.3:41417)
0928 05:21:34.762430 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:34.762509 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":210,"threads_started":1}
I20260928 05:21:34.919116 17452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:34.918354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32969 (local address 127.0.97.1:47865)
0928 05:21:34.918849 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:34.918934 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":289,"threads_started":1}
I20260928 05:21:35.337131 17453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:35.336410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32969 (local address 127.0.97.2:47109)
0928 05:21:35.336839 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:35.336981 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":210,"threads_started":1}
I20260928 05:21:35.764118 17454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:35.763450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32969 (local address 127.0.97.3:47193)
0928 05:21:35.763884 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:35.763967 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":194,"threads_started":1}
I20260928 05:21:35.920785 17455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:35.920070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32969 (local address 127.0.97.1:44389)
0928 05:21:35.920527 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:35.920608 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":227,"threads_started":1}
I20260928 05:21:36.338688 17456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:36.337962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32969 (local address 127.0.97.2:42829)
0928 05:21:36.338436 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:36.338514 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":243,"threads_started":1}
I20260928 05:21:36.765642 17457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:36.764984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32969 (local address 127.0.97.3:45437)
0928 05:21:36.765397 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:36.765473 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":221,"threads_started":1}
I20260928 05:21:36.922206 17458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:36.921572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32969 (local address 127.0.97.1:57931)
0928 05:21:36.921958 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:36.922033 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":191,"threads_started":1}
I20260928 05:21:37.316461 17459 raft_consensus.cc:482] T 196ffebfe3f548ff8bf8674ab4773f25 P 3457ee2e0bfb450cbf6b432ce660d3b9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:21:37.316644 17459 raft_consensus.cc:504] T 196ffebfe3f548ff8bf8674ab4773f25 P 3457ee2e0bfb450cbf6b432ce660d3b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f265296eba24246a0acd818360338ef" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44643 } } peers { permanent_uuid: "3457ee2e0bfb450cbf6b432ce660d3b9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41017 } } peers { permanent_uuid: "4393522a68d64a33b6a85780a76e8e82" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36609 } }
I20260928 05:21:37.317399 17459 leader_election.cc:288] T 196ffebfe3f548ff8bf8674ab4773f25 P 3457ee2e0bfb450cbf6b432ce660d3b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6f265296eba24246a0acd818360338ef (127.0.97.2:44643), 4393522a68d64a33b6a85780a76e8e82 (127.0.97.3:36609)
W20260928 05:21:37.318269 17042 leader_election.cc:341] T 196ffebfe3f548ff8bf8674ab4773f25 P 3457ee2e0bfb450cbf6b432ce660d3b9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4393522a68d64a33b6a85780a76e8e82 (127.0.97.3:36609): Not found: Tablet not found: 196ffebfe3f548ff8bf8674ab4773f25
W20260928 05:21:37.318449 17040 leader_election.cc:341] T 196ffebfe3f548ff8bf8674ab4773f25 P 3457ee2e0bfb450cbf6b432ce660d3b9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6f265296eba24246a0acd818360338ef (127.0.97.2:44643): Not found: Tablet not found: 196ffebfe3f548ff8bf8674ab4773f25
I20260928 05:21:37.318583 17040 leader_election.cc:302] T 196ffebfe3f548ff8bf8674ab4773f25 P 3457ee2e0bfb450cbf6b432ce660d3b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3457ee2e0bfb450cbf6b432ce660d3b9; no voters: 4393522a68d64a33b6a85780a76e8e82, 6f265296eba24246a0acd818360338ef
I20260928 05:21:37.318801 17459 raft_consensus.cc:2726] T 196ffebfe3f548ff8bf8674ab4773f25 P 3457ee2e0bfb450cbf6b432ce660d3b9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 05:21:37.340145 17460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:37.339465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32969 (local address 127.0.97.2:34105)
0928 05:21:37.339906 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:37.339986 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":213,"threads_started":1}
I20260928 05:21:37.767081 17461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:37.766399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32969 (local address 127.0.97.3:48683)
0928 05:21:37.766852 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:37.766934 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":233,"threads_started":1}
I20260928 05:21:37.923616 17462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:37.923004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32969 (local address 127.0.97.1:42139)
0928 05:21:37.923387 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:37.923461 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":202,"threads_started":1}
I20260928 05:21:38.341576 17463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:38.340889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32969 (local address 127.0.97.2:60927)
0928 05:21:38.341308 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:38.341387 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":217,"threads_started":1}
I20260928 05:21:38.768435 17464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:38.767815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32969 (local address 127.0.97.3:57491)
0928 05:21:38.768201 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:38.768273 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":204,"threads_started":1}
I20260928 05:21:38.925143 17465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:38.924413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32969 (local address 127.0.97.1:48415)
0928 05:21:38.924844 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:38.924946 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":200,"threads_started":1}
I20260928 05:21:38.993194 17466 raft_consensus.cc:482] T 196ffebfe3f548ff8bf8674ab4773f25 P 3457ee2e0bfb450cbf6b432ce660d3b9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:21:38.993396 17466 raft_consensus.cc:504] T 196ffebfe3f548ff8bf8674ab4773f25 P 3457ee2e0bfb450cbf6b432ce660d3b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f265296eba24246a0acd818360338ef" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44643 } } peers { permanent_uuid: "3457ee2e0bfb450cbf6b432ce660d3b9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41017 } } peers { permanent_uuid: "4393522a68d64a33b6a85780a76e8e82" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36609 } }
I20260928 05:21:38.994006 17466 leader_election.cc:288] T 196ffebfe3f548ff8bf8674ab4773f25 P 3457ee2e0bfb450cbf6b432ce660d3b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6f265296eba24246a0acd818360338ef (127.0.97.2:44643), 4393522a68d64a33b6a85780a76e8e82 (127.0.97.3:36609)
W20260928 05:21:38.994869 17040 leader_election.cc:341] T 196ffebfe3f548ff8bf8674ab4773f25 P 3457ee2e0bfb450cbf6b432ce660d3b9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6f265296eba24246a0acd818360338ef (127.0.97.2:44643): Not found: Tablet not found: 196ffebfe3f548ff8bf8674ab4773f25
W20260928 05:21:38.995018 17042 leader_election.cc:341] T 196ffebfe3f548ff8bf8674ab4773f25 P 3457ee2e0bfb450cbf6b432ce660d3b9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4393522a68d64a33b6a85780a76e8e82 (127.0.97.3:36609): Not found: Tablet not found: 196ffebfe3f548ff8bf8674ab4773f25
I20260928 05:21:38.995148 17042 leader_election.cc:302] T 196ffebfe3f548ff8bf8674ab4773f25 P 3457ee2e0bfb450cbf6b432ce660d3b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3457ee2e0bfb450cbf6b432ce660d3b9; no voters: 4393522a68d64a33b6a85780a76e8e82, 6f265296eba24246a0acd818360338ef
I20260928 05:21:38.995364 17466 raft_consensus.cc:2726] T 196ffebfe3f548ff8bf8674ab4773f25 P 3457ee2e0bfb450cbf6b432ce660d3b9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 05:21:39.343017 17467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:39.342342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32969 (local address 127.0.97.2:48461)
0928 05:21:39.342755 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:39.342838 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":210,"threads_started":1}
I20260928 05:21:39.769832 17468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:39.769183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32969 (local address 127.0.97.3:34609)
0928 05:21:39.769603 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:39.769678 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":213,"threads_started":1}
I20260928 05:21:39.926545 17469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:39.925884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32969 (local address 127.0.97.1:59721)
0928 05:21:39.926288 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:39.926364 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":209,"threads_started":1}
I20260928 05:21:40.344388 17470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:40.343755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32969 (local address 127.0.97.2:40051)
0928 05:21:40.344133 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:40.344207 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":190,"threads_started":1}
I20260928 05:21:40.771337 17471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:40.770630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32969 (local address 127.0.97.3:39461)
0928 05:21:40.771057 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:40.771145 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":218,"threads_started":1}
I20260928 05:21:40.928370 17472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:40.927657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32969 (local address 127.0.97.1:55725)
0928 05:21:40.928114 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:40.928195 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":216,"threads_started":1}
I20260928 05:21:41.345983 17473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:41.345364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32969 (local address 127.0.97.2:50633)
0928 05:21:41.345737 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:41.345819 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":185,"threads_started":1}
I20260928 05:21:41.754056 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 16984
I20260928 05:21:41.767576 17474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:41.754998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49210 (local address 127.0.97.1:41017)
0928 05:21:41.755611 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0928 05:21:41.755621 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 05:21:41.755655 (+ 34us) server_negotiation.cc:407] Connection header received
0928 05:21:41.755736 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:21:41.755746 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:21:41.756012 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:21:41.756139 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:21:41.757590 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:41.758551 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:41.759702 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:41.760224 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:41.760313 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:41.762589 (+ 2276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:21:41.762599 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:21:41.762613 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:21:41.762647 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:21:41.765387 (+ 2740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:21:41.765971 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:21:41.765977 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:21:41.765979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:21:41.766096 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:21:41.766357 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:21:41.766359 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:21:41.766361 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:21:41.766681 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:21:41.766901 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0928 05:21:41.767081 (+ 180us) server_negotiation.cc:299] Negotiation successful
0928 05:21:41.767259 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":287,"threads_started":1}
I20260928 05:21:41.768683 17149 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:21:41.773119 17476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:41.772397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32969 (local address 127.0.97.3:43805)
0928 05:21:41.772801 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:41.772914 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":210,"threads_started":1}
I20260928 05:21:41.856632 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 16984
I20260928 05:21:41.871101 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 17157
I20260928 05:21:41.883482 17477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:41.871827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38678 (local address 127.0.97.2:44643)
0928 05:21:41.872362 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0928 05:21:41.872366 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:21:41.872382 (+ 16us) server_negotiation.cc:407] Connection header received
0928 05:21:41.872448 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:21:41.872451 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:21:41.872509 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:21:41.872596 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:21:41.873883 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:41.874691 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:41.875875 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:41.876299 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:41.876393 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:41.878753 (+ 2360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:21:41.878758 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:21:41.878761 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:21:41.878792 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:21:41.881272 (+ 2480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:21:41.881872 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:21:41.881875 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:21:41.881877 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:21:41.882007 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:21:41.882299 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:21:41.882302 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:21:41.882304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:21:41.882544 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:21:41.882762 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0928 05:21:41.882957 (+ 195us) server_negotiation.cc:299] Negotiation successful
0928 05:21:41.883139 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":316,"threads_started":1}
I20260928 05:21:41.884143 17281 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:21:41.964017 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 17157
I20260928 05:21:41.978320 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 17289
I20260928 05:21:41.990724 17479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:41.979135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37786 (local address 127.0.97.3:36609)
0928 05:21:41.979643 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0928 05:21:41.979647 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:21:41.979665 (+ 18us) server_negotiation.cc:407] Connection header received
0928 05:21:41.979719 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:21:41.979723 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:21:41.979782 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:21:41.979871 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:21:41.981058 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:41.981876 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:41.983128 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:41.983588 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:41.983682 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:41.986105 (+ 2423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:21:41.986109 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:21:41.986112 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:21:41.986145 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:21:41.988594 (+ 2449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:21:41.989214 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:21:41.989218 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:21:41.989220 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:21:41.989341 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:21:41.989609 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:21:41.989612 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:21:41.989614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:21:41.989848 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:21:41.990055 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0928 05:21:41.990226 (+ 171us) server_negotiation.cc:299] Negotiation successful
0928 05:21:41.990372 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":242,"threads_started":1}
I20260928 05:21:41.991386 17412 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:21:42.074565 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 17289
2026-09-28T05:21:42Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (63616 ms)
[----------] 30 tests from SecurityITest (279459 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572622576160-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:21:42 dist-test-slave-whsn krb5kdc[17483](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:21:42 dist-test-slave-whsn krb5kdc[17483](info): set up 2 sockets
Sep 28 05:21:42 dist-test-slave-whsn krb5kdc[17483](info): commencing operation
krb5kdc: starting...
W20260928 05:21:43.080593 388 mini_kdc.cc:121] Time spent starting KDC: real 0.971s user 0.003s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572622576160-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572622576160-388-0/krb5kdc/test-user.keytab.
Sep 28 05:21:43 dist-test-slave-whsn krb5kdc[17483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572903, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:21:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:21:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 05:21:43.286731 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572622576160-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572622576160-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572622576160-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45091
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45091
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:21:43.590962 17496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:21:43.591216 17496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:21:43.591301 17496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:21:43.598191 17496 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:21:43.598278 17496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:21:43.598320 17496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:21:43.598373 17496 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:21:43.598415 17496 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:21:43.607666 17496 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572622576160-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572622576160-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45091
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572622576160-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:21:43.608814 17496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:21:43.610088 17496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:21:43.621018 17502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:21:43.621069 17504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:21:43.621132 17501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:21:43.621354 17496 server_base.cc:1034] running on GCE node
I20260928 05:21:43.622184 17496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:21:43.623436 17496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:21:43.624629 17496 hybrid_clock.cc:648] HybridClock initialized: now 1790572903624585 us; error 29 us; skew 500 ppm
I20260928 05:21:43.624974 17496 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:21:43 dist-test-slave-whsn krb5kdc[17483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572903, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:21:43.627779 17496 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 05:21:43.629103 17496 webserver.cc:460] Webserver started at http://127.0.97.62:37577/ using document root <none> and password file <none>
I20260928 05:21:43.629720 17496 fs_manager.cc:359] Metadata directory not provided
I20260928 05:21:43.629832 17496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:21:43.630098 17496 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:21:43.632774 17496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572622576160-388-0/minicluster-data/master-0/data/instance:
uuid: "de8ad57463bc4b5aba2224dfb24bd193"
format_stamp: "Formatted at 2026-09-28 05:21:43 on dist-test-slave-whsn"
server_key: "139044c2e248bfe21bc85a0dbd6b7c1a"
server_key_iv: "e079477a9ee37629c4557dcfee9a83ab"
server_key_version: "encryptionkey@0"
I20260928 05:21:43.633492 17496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572622576160-388-0/minicluster-data/master-0/wal/instance:
uuid: "de8ad57463bc4b5aba2224dfb24bd193"
format_stamp: "Formatted at 2026-09-28 05:21:43 on dist-test-slave-whsn"
server_key: "139044c2e248bfe21bc85a0dbd6b7c1a"
server_key_iv: "e079477a9ee37629c4557dcfee9a83ab"
server_key_version: "encryptionkey@0"
I20260928 05:21:43.638119 17496 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260928 05:21:43.641237 17511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:21:43.642400 17496 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.002s
I20260928 05:21:43.642565 17496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572622576160-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572622576160-388-0/minicluster-data/master-0/wal
uuid: "de8ad57463bc4b5aba2224dfb24bd193"
format_stamp: "Formatted at 2026-09-28 05:21:43 on dist-test-slave-whsn"
server_key: "139044c2e248bfe21bc85a0dbd6b7c1a"
server_key_iv: "e079477a9ee37629c4557dcfee9a83ab"
server_key_version: "encryptionkey@0"
I20260928 05:21:43.642724 17496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572622576160-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572622576160-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572622576160-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:21:43.658663 17496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:21:43.662385 17496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:21:43.662606 17496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:21:43.678581 17496 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45091
I20260928 05:21:43.678622 17562 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45091 every 8 connection(s)
I20260928 05:21:43.679826 17496 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:21:43.683525 17563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:21:43.685246 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 17496
I20260928 05:21:43.685447 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572622576160-388-0/minicluster-data/master-0/wal/instance
I20260928 05:21:43.686100 388 external_mini_cluster.cc:1428] Setting key 39ba6ee8c86295c831e2702797415630
I20260928 05:21:43.695525 17563 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de8ad57463bc4b5aba2224dfb24bd193: Bootstrap starting.
I20260928 05:21:43.698722 17563 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de8ad57463bc4b5aba2224dfb24bd193: Neither blocks nor log segments found. Creating new log.
I20260928 05:21:43.699635 17563 log.cc:824] T 00000000000000000000000000000000 P de8ad57463bc4b5aba2224dfb24bd193: Log is configured to *not* fsync() on all Append() calls
Sep 28 05:21:43 dist-test-slave-whsn krb5kdc[17483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572903, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:21:43.702404 17563 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de8ad57463bc4b5aba2224dfb24bd193: No bootstrap required, opened a new log
I20260928 05:21:43.709826 17566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:43.688061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54652 (local address 127.0.97.62:45091)
0928 05:21:43.688889 (+ 828us) server_negotiation.cc:206] Beginning negotiation
0928 05:21:43.688904 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 05:21:43.688953 (+ 49us) server_negotiation.cc:407] Connection header received
0928 05:21:43.691224 (+ 2271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:21:43.691251 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:21:43.691723 (+ 472us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:21:43.692141 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:21:43.693551 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:43.695392 (+ 1841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:43.696829 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:43.697817 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:43.697995 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:43.702444 (+ 4449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:21:43.702455 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:21:43.702474 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:21:43.702527 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:21:43.705755 (+ 3228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:21:43.706459 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:21:43.706466 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:21:43.706472 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:21:43.706657 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:21:43.707073 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:21:43.707077 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:21:43.707079 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:21:43.707655 (+ 576us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:21:43.707937 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0928 05:21:43.708437 (+ 500us) server_negotiation.cc:299] Negotiation successful
0928 05:21:43.708825 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":329,"threads_started":1}
I20260928 05:21:43.709789 17563 raft_consensus.cc:348] T 00000000000000000000000000000000 P de8ad57463bc4b5aba2224dfb24bd193 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de8ad57463bc4b5aba2224dfb24bd193" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45091 } }
I20260928 05:21:43.710193 17563 raft_consensus.cc:374] T 00000000000000000000000000000000 P de8ad57463bc4b5aba2224dfb24bd193 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:21:43.710289 17563 raft_consensus.cc:729] T 00000000000000000000000000000000 P de8ad57463bc4b5aba2224dfb24bd193 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de8ad57463bc4b5aba2224dfb24bd193, State: Initialized, Role: FOLLOWER
I20260928 05:21:43.710839 17563 consensus_queue.cc:260] T 00000000000000000000000000000000 P de8ad57463bc4b5aba2224dfb24bd193 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de8ad57463bc4b5aba2224dfb24bd193" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45091 } }
I20260928 05:21:43.711064 17563 raft_consensus.cc:388] T 00000000000000000000000000000000 P de8ad57463bc4b5aba2224dfb24bd193 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:21:43.711205 17563 raft_consensus.cc:482] T 00000000000000000000000000000000 P de8ad57463bc4b5aba2224dfb24bd193 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:21:43.711373 17563 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de8ad57463bc4b5aba2224dfb24bd193 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:21:43.713907 17563 raft_consensus.cc:504] T 00000000000000000000000000000000 P de8ad57463bc4b5aba2224dfb24bd193 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de8ad57463bc4b5aba2224dfb24bd193" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45091 } }
I20260928 05:21:43.714411 17563 leader_election.cc:302] T 00000000000000000000000000000000 P de8ad57463bc4b5aba2224dfb24bd193 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de8ad57463bc4b5aba2224dfb24bd193; no voters:
I20260928 05:21:43.714996 17563 leader_election.cc:288] T 00000000000000000000000000000000 P de8ad57463bc4b5aba2224dfb24bd193 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:21:43.715072 17568 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de8ad57463bc4b5aba2224dfb24bd193 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:21:43.715700 17568 raft_consensus.cc:686] T 00000000000000000000000000000000 P de8ad57463bc4b5aba2224dfb24bd193 [term 1 LEADER]: Becoming Leader. State: Replica: de8ad57463bc4b5aba2224dfb24bd193, State: Running, Role: LEADER
I20260928 05:21:43.716204 17568 consensus_queue.cc:237] T 00000000000000000000000000000000 P de8ad57463bc4b5aba2224dfb24bd193 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de8ad57463bc4b5aba2224dfb24bd193" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45091 } }
I20260928 05:21:43.716743 17563 sys_catalog.cc:564] T 00000000000000000000000000000000 P de8ad57463bc4b5aba2224dfb24bd193 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:21:43.718925 17569 sys_catalog.cc:455] T 00000000000000000000000000000000 P de8ad57463bc4b5aba2224dfb24bd193 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "de8ad57463bc4b5aba2224dfb24bd193" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de8ad57463bc4b5aba2224dfb24bd193" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45091 } } }
I20260928 05:21:43.719281 17569 sys_catalog.cc:458] T 00000000000000000000000000000000 P de8ad57463bc4b5aba2224dfb24bd193 [sys.catalog]: This master's current role is: LEADER
I20260928 05:21:43.719924 17570 sys_catalog.cc:455] T 00000000000000000000000000000000 P de8ad57463bc4b5aba2224dfb24bd193 [sys.catalog]: SysCatalogTable state changed. Reason: New leader de8ad57463bc4b5aba2224dfb24bd193. Latest consensus state: current_term: 1 leader_uuid: "de8ad57463bc4b5aba2224dfb24bd193" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de8ad57463bc4b5aba2224dfb24bd193" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45091 } } }
I20260928 05:21:43.720221 17570 sys_catalog.cc:458] T 00000000000000000000000000000000 P de8ad57463bc4b5aba2224dfb24bd193 [sys.catalog]: This master's current role is: LEADER
I20260928 05:21:43.721866 17575 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:21:43.726656 17575 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:21:43.734730 17575 catalog_manager.cc:1347] Generated new cluster ID: 8ea47d0bb5c34cea8809ef123d586318
I20260928 05:21:43.734817 17575 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:21:43.754542 17575 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:21:43.756314 17575 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:21:43.762925 17575 catalog_manager.cc:5868] T 00000000000000000000000000000000 P de8ad57463bc4b5aba2224dfb24bd193: Generated new TSK 0
I20260928 05:21:43.763899 17575 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572622576160-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572622576160-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260928 05:21:43.821358 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572622576160-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572622576160-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572622576160-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572622576160-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45091
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572622576160-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:21:44.126286 17589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:21:44.126542 17589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:21:44.126679 17589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:21:44.133628 17589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:21:44.133715 17589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:21:44.133848 17589 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 05:21:44.143093 17589 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572622576160-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572622576160-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572622576160-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572622576160-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45091
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572622576160-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:21:44.144258 17589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:21:44.145507 17589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:21:44.157698 17595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:21:44.157760 17597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:21:44.157790 17594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:21:44.158346 17589 server_base.cc:1034] running on GCE node
I20260928 05:21:44.159566 17589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:21:44.160477 17589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:21:44.161686 17589 hybrid_clock.cc:648] HybridClock initialized: now 1790572904161635 us; error 46 us; skew 500 ppm
I20260928 05:21:44.162019 17589 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:21:44 dist-test-slave-whsn krb5kdc[17483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572904, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:21:44.164978 17589 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260928 05:21:44.166409 17589 webserver.cc:460] Webserver started at http://127.0.97.1:46827/ using document root <none> and password file <none>
I20260928 05:21:44.167052 17589 fs_manager.cc:359] Metadata directory not provided
I20260928 05:21:44.167158 17589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:21:44.167429 17589 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:21:44.170233 17589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572622576160-388-0/minicluster-data/ts-0/data/instance:
uuid: "e8b12dcaf16d4483aa41523ee70b768f"
format_stamp: "Formatted at 2026-09-28 05:21:44 on dist-test-slave-whsn"
server_key: "3386ca34ce3ab0b440ce33cfa429163e"
server_key_iv: "6ffbd90997e37296d4e3753d1583d6b8"
server_key_version: "encryptionkey@0"
I20260928 05:21:44.170931 17589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572622576160-388-0/minicluster-data/ts-0/wal/instance:
uuid: "e8b12dcaf16d4483aa41523ee70b768f"
format_stamp: "Formatted at 2026-09-28 05:21:44 on dist-test-slave-whsn"
server_key: "3386ca34ce3ab0b440ce33cfa429163e"
server_key_iv: "6ffbd90997e37296d4e3753d1583d6b8"
server_key_version: "encryptionkey@0"
I20260928 05:21:44.175590 17589 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 05:21:44.178745 17604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:21:44.179926 17589 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260928 05:21:44.180075 17589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572622576160-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572622576160-388-0/minicluster-data/ts-0/wal
uuid: "e8b12dcaf16d4483aa41523ee70b768f"
format_stamp: "Formatted at 2026-09-28 05:21:44 on dist-test-slave-whsn"
server_key: "3386ca34ce3ab0b440ce33cfa429163e"
server_key_iv: "6ffbd90997e37296d4e3753d1583d6b8"
server_key_version: "encryptionkey@0"
I20260928 05:21:44.180238 17589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572622576160-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572622576160-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572622576160-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:21:44.198061 17589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:21:44.201848 17589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:21:44.202096 17589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:21:44.202927 17589 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:21:44.204116 17589 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:21:44.204187 17589 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:21:44.204304 17589 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:21:44.204355 17589 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:21:44.232820 17589 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38815
I20260928 05:21:44.232890 17716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38815 every 8 connection(s)
I20260928 05:21:44.234014 17589 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572622576160-388-0/minicluster-data/ts-0/data/info.pb
I20260928 05:21:44.241006 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 17589
I20260928 05:21:44.241171 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572622576160-388-0/minicluster-data/ts-0/wal/instance
I20260928 05:21:44.241778 388 external_mini_cluster.cc:1428] Setting key 19ace01ee4109a9e6ae419e58e033c14
Sep 28 05:21:44 dist-test-slave-whsn krb5kdc[17483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572904, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:21:44.252554 17719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:44.236253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55263 (local address 127.0.97.62:45091)
0928 05:21:44.236647 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0928 05:21:44.236651 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:21:44.237628 (+ 977us) server_negotiation.cc:407] Connection header received
0928 05:21:44.239282 (+ 1654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:21:44.239286 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:21:44.239351 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:21:44.239441 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:21:44.240840 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:44.241633 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:44.243379 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:44.243832 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:44.243923 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:44.247468 (+ 3545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:21:44.247472 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:21:44.247476 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:21:44.247506 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:21:44.249850 (+ 2344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:21:44.250474 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:21:44.250480 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:21:44.250484 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:21:44.250597 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:21:44.250967 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:21:44.250972 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:21:44.250975 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:21:44.251245 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:21:44.251373 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 05:21:44.252001 (+ 628us) server_negotiation.cc:299] Negotiation successful
0928 05:21:44.252209 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":195,"threads_started":1}
I20260928 05:21:44.253439 17720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:44.236792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45091 (local address 127.0.97.1:55263)
0928 05:21:44.237472 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:44.237515 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 05:21:44.239061 (+ 1546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:21:44.239647 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:21:44.239655 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:21:44.240112 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:21:44.240677 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:44.240701 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:44.241769 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:44.241775 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:44.243240 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:44.243254 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:44.244044 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:44.244052 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:44.244317 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:44.244933 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:21:44.244964 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:21:44.247276 (+ 2312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:21:44.249973 (+ 2697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:21:44.249981 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:21:44.249995 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:21:44.250368 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:21:44.250695 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:21:44.250700 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:21:44.250704 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:21:44.250869 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:21:44.251362 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:21:44.251369 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:21:44.251668 (+ 299us) client_negotiation.cc:769] Sending connection context
0928 05:21:44.251904 (+ 236us) client_negotiation.cc:240] Negotiation successful
0928 05:21:44.252361 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":249,"threads_started":1}
I20260928 05:21:44.254848 17717 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45091
I20260928 05:21:44.255118 17717 heartbeater.cc:464] Registering TS with master...
I20260928 05:21:44.255751 17717 heartbeater.cc:511] Master 127.0.97.62:45091 requested a full tablet report, sending...
I20260928 05:21:44.257824 17528 ts_manager.cc:194] Registered new tserver with Master: e8b12dcaf16d4483aa41523ee70b768f (127.0.97.1:38815)
I20260928 05:21:44.259728 17528 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:55263
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572622576160-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572622576160-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260928 05:21:44.287966 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572622576160-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572622576160-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572622576160-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572622576160-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45091
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572622576160-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:21:44.590826 17723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:21:44.591075 17723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:21:44.591192 17723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:21:44.598183 17723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:21:44.598281 17723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:21:44.598426 17723 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 05:21:44.607748 17723 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572622576160-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572622576160-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572622576160-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572622576160-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45091
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572622576160-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:21:44.608932 17723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:21:44.610160 17723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:21:44.622069 17728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:21:44.622071 17729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:21:44.622298 17731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:21:44.623179 17723 server_base.cc:1034] running on GCE node
I20260928 05:21:44.623725 17723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:21:44.624555 17723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:21:44.625736 17723 hybrid_clock.cc:648] HybridClock initialized: now 1790572904625690 us; error 41 us; skew 500 ppm
I20260928 05:21:44.626040 17723 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:21:44 dist-test-slave-whsn krb5kdc[17483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572904, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:21:44.629136 17723 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260928 05:21:44.630436 17723 webserver.cc:460] Webserver started at http://127.0.97.2:35531/ using document root <none> and password file <none>
I20260928 05:21:44.631024 17723 fs_manager.cc:359] Metadata directory not provided
I20260928 05:21:44.631115 17723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:21:44.631347 17723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:21:44.634061 17723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572622576160-388-0/minicluster-data/ts-1/data/instance:
uuid: "0b36a61cea5c429a8c9621ccb934aa08"
format_stamp: "Formatted at 2026-09-28 05:21:44 on dist-test-slave-whsn"
server_key: "6b17fc910df583874d768c28a0e3c9e0"
server_key_iv: "821f9764b058f78b7b9a02aae7e22eb6"
server_key_version: "encryptionkey@0"
I20260928 05:21:44.634739 17723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572622576160-388-0/minicluster-data/ts-1/wal/instance:
uuid: "0b36a61cea5c429a8c9621ccb934aa08"
format_stamp: "Formatted at 2026-09-28 05:21:44 on dist-test-slave-whsn"
server_key: "6b17fc910df583874d768c28a0e3c9e0"
server_key_iv: "821f9764b058f78b7b9a02aae7e22eb6"
server_key_version: "encryptionkey@0"
I20260928 05:21:44.639312 17723 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 05:21:44.642465 17738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:21:44.643690 17723 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 05:21:44.643808 17723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572622576160-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572622576160-388-0/minicluster-data/ts-1/wal
uuid: "0b36a61cea5c429a8c9621ccb934aa08"
format_stamp: "Formatted at 2026-09-28 05:21:44 on dist-test-slave-whsn"
server_key: "6b17fc910df583874d768c28a0e3c9e0"
server_key_iv: "821f9764b058f78b7b9a02aae7e22eb6"
server_key_version: "encryptionkey@0"
I20260928 05:21:44.643956 17723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572622576160-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572622576160-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572622576160-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:21:44.658253 17723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:21:44.662004 17723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:21:44.662281 17723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:21:44.663107 17723 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:21:44.664248 17723 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:21:44.664318 17723 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:21:44.664402 17723 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:21:44.664445 17723 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:21:44.693954 17723 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43377
I20260928 05:21:44.694029 17850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43377 every 8 connection(s)
I20260928 05:21:44.695106 17723 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572622576160-388-0/minicluster-data/ts-1/data/info.pb
I20260928 05:21:44.697149 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 17723
I20260928 05:21:44.697322 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572622576160-388-0/minicluster-data/ts-1/wal/instance
I20260928 05:21:44.697728 388 external_mini_cluster.cc:1428] Setting key 413dd6bb27dfa9ad675ca6028ac9e3ca
Sep 28 05:21:44 dist-test-slave-whsn krb5kdc[17483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572904, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:21:44.712888 17719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:44.697343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57699 (local address 127.0.97.62:45091)
0928 05:21:44.697533 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 05:21:44.697538 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:21:44.698710 (+ 1172us) server_negotiation.cc:407] Connection header received
0928 05:21:44.700456 (+ 1746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:21:44.700461 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:21:44.700545 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:21:44.700636 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:21:44.702030 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:44.702727 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:44.704166 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:44.704555 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:44.704669 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:44.707897 (+ 3228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:21:44.707902 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:21:44.707905 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:21:44.707935 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:21:44.710229 (+ 2294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:21:44.710833 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:21:44.710838 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:21:44.710840 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:21:44.710977 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:21:44.711352 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:21:44.711355 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:21:44.711357 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:21:44.711595 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:21:44.711730 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 05:21:44.712329 (+ 599us) server_negotiation.cc:299] Negotiation successful
0928 05:21:44.712543 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 05:21:44.713752 17853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:44.697815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45091 (local address 127.0.97.2:57699)
0928 05:21:44.698566 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:44.698608 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 05:21:44.700213 (+ 1605us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:21:44.700874 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:21:44.700887 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:21:44.701367 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:21:44.701880 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:44.701895 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:44.702871 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:44.702875 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:44.704033 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:44.704044 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:44.704750 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:44.704758 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:44.705017 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:44.705585 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:21:44.705605 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:21:44.707711 (+ 2106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:21:44.710346 (+ 2635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:21:44.710354 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:21:44.710368 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:21:44.710730 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:21:44.711073 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:21:44.711078 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:21:44.711082 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:21:44.711255 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:21:44.711716 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:21:44.711723 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:21:44.712005 (+ 282us) client_negotiation.cc:769] Sending connection context
0928 05:21:44.712228 (+ 223us) client_negotiation.cc:240] Negotiation successful
0928 05:21:44.712675 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":302,"threads_started":1}
I20260928 05:21:44.714989 17851 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45091
I20260928 05:21:44.715248 17851 heartbeater.cc:464] Registering TS with master...
I20260928 05:21:44.715878 17851 heartbeater.cc:511] Master 127.0.97.62:45091 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260928 05:21:44.717200 17528 ts_manager.cc:194] Registered new tserver with Master: 0b36a61cea5c429a8c9621ccb934aa08 (127.0.97.2:43377)
I20260928 05:21:44.718021 17528 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:57699
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572622576160-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572622576160-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260928 05:21:44.741863 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572622576160-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572622576160-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572622576160-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572622576160-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45091
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572622576160-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:21:45.041579 17856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:21:45.041826 17856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:21:45.041915 17856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:21:45.049000 17856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:21:45.049090 17856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:21:45.049202 17856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 05:21:45.058272 17856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572622576160-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572622576160-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572622576160-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572622576160-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45091
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572622576160-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:21:45.059398 17856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:21:45.061087 17856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:21:45.074410 17861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:21:45.074286 17862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:21:45.074329 17864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:21:45.075031 17856 server_base.cc:1034] running on GCE node
I20260928 05:21:45.075660 17856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:21:45.076604 17856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:21:45.077790 17856 hybrid_clock.cc:648] HybridClock initialized: now 1790572905077733 us; error 56 us; skew 500 ppm
I20260928 05:21:45.078099 17856 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:21:45 dist-test-slave-whsn krb5kdc[17483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572905, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:21:45.081041 17856 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260928 05:21:45.082382 17856 webserver.cc:460] Webserver started at http://127.0.97.3:46107/ using document root <none> and password file <none>
I20260928 05:21:45.083055 17856 fs_manager.cc:359] Metadata directory not provided
I20260928 05:21:45.083169 17856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:21:45.083427 17856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:21:45.086277 17856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572622576160-388-0/minicluster-data/ts-2/data/instance:
uuid: "fb7f98e62a7e4cfcabb04656fc6d03a0"
format_stamp: "Formatted at 2026-09-28 05:21:45 on dist-test-slave-whsn"
server_key: "16996aae9372211b6c53cb7f4a57e9a3"
server_key_iv: "e488b7cdbdb7a971ed1c8c1e119298ab"
server_key_version: "encryptionkey@0"
I20260928 05:21:45.086980 17856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572622576160-388-0/minicluster-data/ts-2/wal/instance:
uuid: "fb7f98e62a7e4cfcabb04656fc6d03a0"
format_stamp: "Formatted at 2026-09-28 05:21:45 on dist-test-slave-whsn"
server_key: "16996aae9372211b6c53cb7f4a57e9a3"
server_key_iv: "e488b7cdbdb7a971ed1c8c1e119298ab"
server_key_version: "encryptionkey@0"
I20260928 05:21:45.091711 17856 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 05:21:45.095059 17871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:21:45.096407 17856 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 05:21:45.096544 17856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572622576160-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572622576160-388-0/minicluster-data/ts-2/wal
uuid: "fb7f98e62a7e4cfcabb04656fc6d03a0"
format_stamp: "Formatted at 2026-09-28 05:21:45 on dist-test-slave-whsn"
server_key: "16996aae9372211b6c53cb7f4a57e9a3"
server_key_iv: "e488b7cdbdb7a971ed1c8c1e119298ab"
server_key_version: "encryptionkey@0"
I20260928 05:21:45.096709 17856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572622576160-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572622576160-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572622576160-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:21:45.117686 17856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:21:45.121798 17856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:21:45.122047 17856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:21:45.123018 17856 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:21:45.124253 17856 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:21:45.124325 17856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:21:45.124439 17856 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:21:45.124496 17856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:21:45.153354 17856 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44193
I20260928 05:21:45.153396 17983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44193 every 8 connection(s)
I20260928 05:21:45.154625 17856 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572622576160-388-0/minicluster-data/ts-2/data/info.pb
I20260928 05:21:45.159070 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 17856
I20260928 05:21:45.159245 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572622576160-388-0/minicluster-data/ts-2/wal/instance
I20260928 05:21:45.159785 388 external_mini_cluster.cc:1428] Setting key 3cb34084b9580b314679e155607dc389
Sep 28 05:21:45 dist-test-slave-whsn krb5kdc[17483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572905, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:21:45.174315 17719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:45.157062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38079 (local address 127.0.97.62:45091)
0928 05:21:45.157262 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 05:21:45.157266 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:21:45.158404 (+ 1138us) server_negotiation.cc:407] Connection header received
0928 05:21:45.160716 (+ 2312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:21:45.160722 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:21:45.160809 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:21:45.160947 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:21:45.162712 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:45.163477 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:45.164978 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:45.165390 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:45.165483 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:45.169000 (+ 3517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:21:45.169018 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:21:45.169021 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:21:45.169053 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:21:45.171397 (+ 2344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:21:45.172077 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:21:45.172080 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:21:45.172082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:21:45.172189 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:21:45.172574 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:21:45.172577 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:21:45.172579 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:21:45.172814 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:21:45.172970 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 05:21:45.173829 (+ 859us) server_negotiation.cc:299] Negotiation successful
0928 05:21:45.174022 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 05:21:45.175436 17986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:45.157555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45091 (local address 127.0.97.3:38079)
0928 05:21:45.158236 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:45.158278 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 05:21:45.160413 (+ 2135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:21:45.161188 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:21:45.161199 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:21:45.161926 (+ 727us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:21:45.162551 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:45.162572 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:45.163613 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:45.163617 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:45.164833 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:45.164844 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:45.165611 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:45.165617 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:45.165827 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:45.166521 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:21:45.166547 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:21:45.168775 (+ 2228us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:21:45.171527 (+ 2752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:21:45.171533 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:21:45.171545 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:21:45.171980 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:21:45.172284 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:21:45.172287 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:21:45.172289 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:21:45.172489 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:21:45.172973 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:21:45.172980 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:21:45.173386 (+ 406us) client_negotiation.cc:769] Sending connection context
0928 05:21:45.173701 (+ 315us) client_negotiation.cc:240] Negotiation successful
0928 05:21:45.174210 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":270,"threads_started":1}
I20260928 05:21:45.176621 17984 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45091
I20260928 05:21:45.176909 17984 heartbeater.cc:464] Registering TS with master...
I20260928 05:21:45.177603 17984 heartbeater.cc:511] Master 127.0.97.62:45091 requested a full tablet report, sending...
I20260928 05:21:45.178836 17528 ts_manager.cc:194] Registered new tserver with Master: fb7f98e62a7e4cfcabb04656fc6d03a0 (127.0.97.3:44193)
I20260928 05:21:45.179600 17528 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:38079
I20260928 05:21:45.186901 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 05:21:45.202315 17719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:45.191109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54654 (local address 127.0.97.62:45091)
0928 05:21:45.191312 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 05:21:45.191319 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:21:45.191617 (+ 298us) server_negotiation.cc:407] Connection header received
0928 05:21:45.191762 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:21:45.191765 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:21:45.191825 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:21:45.191894 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:21:45.193258 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:45.193939 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:45.195180 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:45.195569 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:45.195656 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:45.197810 (+ 2154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:21:45.197815 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:21:45.197818 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:21:45.197851 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:21:45.200071 (+ 2220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:21:45.200733 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:21:45.200737 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:21:45.200739 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:21:45.200871 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:21:45.201165 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:21:45.201168 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:21:45.201170 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:21:45.201453 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:21:45.201621 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0928 05:21:45.201879 (+ 258us) server_negotiation.cc:299] Negotiation successful
0928 05:21:45.202033 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 05:21:45.206017 17528 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54654:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 05:21:45.212733 17528 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 05:21:45.247623 17996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:45.224186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44193 (local address 127.0.0.1:57628)
0928 05:21:45.224878 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:45.224920 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 05:21:45.225081 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:21:45.240385 (+ 15304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:21:45.240392 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:21:45.240424 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:21:45.240752 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:45.240766 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:45.242781 (+ 2015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:45.242787 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:45.245118 (+ 2331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:45.245132 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:45.246866 (+ 1734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:45.246872 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:45.247066 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:45.247116 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 05:21:45.247228 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 05:21:45.247382 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":15049,"threads_started":1}
I20260928 05:21:45.247743 17998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:45.224333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57628 (local address 127.0.97.3:44193)
0928 05:21:45.239880 (+ 15547us) server_negotiation.cc:206] Beginning negotiation
0928 05:21:45.239886 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:21:45.239912 (+ 26us) server_negotiation.cc:407] Connection header received
0928 05:21:45.239990 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:21:45.239995 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:21:45.240187 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:21:45.240351 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:21:45.241003 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:45.242640 (+ 1637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:45.245277 (+ 2637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:45.246698 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:45.246810 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:45.247028 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0928 05:21:45.247334 (+ 306us) server_negotiation.cc:299] Negotiation successful
0928 05:21:45.247466 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15402,"thread_start_us":15252,"threads_started":1}
I20260928 05:21:45.251737 18001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:45.225874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49126 (local address 127.0.97.1:38815)
0928 05:21:45.241108 (+ 15234us) server_negotiation.cc:206] Beginning negotiation
0928 05:21:45.241116 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:21:45.243816 (+ 2700us) server_negotiation.cc:407] Connection header received
0928 05:21:45.244045 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:21:45.244053 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:21:45.244257 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:21:45.244434 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:21:45.245188 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:45.246683 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:45.249867 (+ 3184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:45.250767 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:45.250884 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:45.251111 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0928 05:21:45.251290 (+ 179us) server_negotiation.cc:299] Negotiation successful
0928 05:21:45.251437 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15078,"thread_start_us":15221,"threads_started":1}
I20260928 05:21:45.251746 18000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:45.225645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38815 (local address 127.0.0.1:49126)
0928 05:21:45.243687 (+ 18042us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:45.243705 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 05:21:45.243819 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:21:45.244439 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:21:45.244443 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:21:45.244466 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:21:45.245022 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:45.245033 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:45.247881 (+ 2848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:45.247885 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:45.249709 (+ 1824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:45.249722 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:45.250950 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:45.250956 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:45.251109 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:45.251141 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 05:21:45.251213 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 05:21:45.251328 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17909,"mutex_wait_us":24,"thread_start_us":17765,"threads_started":1}
I20260928 05:21:45.251837 17919 tablet_service.cc:1467] Processing CreateTablet for tablet a600487619334a42810f4fdea0f11725 (DEFAULT_TABLE table=test-table [id=8218d2ff4c9a4944a92f8db5bb95df94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:21:45.253135 17997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:45.224208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43377 (local address 127.0.0.1:56946)
0928 05:21:45.239007 (+ 14799us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:45.239028 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 05:21:45.239158 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:21:45.244730 (+ 5572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:21:45.244735 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:21:45.244758 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:21:45.245252 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:45.245263 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:45.248009 (+ 2746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:45.248014 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:45.249773 (+ 1759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:45.249787 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:45.252389 (+ 2602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:45.252395 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:45.252550 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:45.252581 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 05:21:45.252648 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 05:21:45.252762 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14659,"mutex_wait_us":19,"thread_start_us":799,"threads_started":1}
I20260928 05:21:45.253360 17919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a600487619334a42810f4fdea0f11725. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:21:45.254782 17652 tablet_service.cc:1467] Processing CreateTablet for tablet a600487619334a42810f4fdea0f11725 (DEFAULT_TABLE table=test-table [id=8218d2ff4c9a4944a92f8db5bb95df94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:21:45.255811 17652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a600487619334a42810f4fdea0f11725. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:21:45.256778 17999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:45.224712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56946 (local address 127.0.97.2:43377)
0928 05:21:45.244224 (+ 19512us) server_negotiation.cc:206] Beginning negotiation
0928 05:21:45.244230 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:21:45.244251 (+ 21us) server_negotiation.cc:407] Connection header received
0928 05:21:45.244320 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:21:45.244324 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:21:45.244534 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:21:45.244728 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:21:45.245391 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:45.247145 (+ 1754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:45.249909 (+ 2764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:45.251207 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:45.256076 (+ 4869us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:45.256269 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0928 05:21:45.256336 (+ 67us) server_negotiation.cc:299] Negotiation successful
0928 05:21:45.256506 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19377,"thread_start_us":19289,"threads_started":1}
I20260928 05:21:45.259975 17786 tablet_service.cc:1467] Processing CreateTablet for tablet a600487619334a42810f4fdea0f11725 (DEFAULT_TABLE table=test-table [id=8218d2ff4c9a4944a92f8db5bb95df94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:21:45.261241 17786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a600487619334a42810f4fdea0f11725. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:21:45.262612 17717 heartbeater.cc:503] Master 127.0.97.62:45091 was elected leader, sending a full tablet report...
I20260928 05:21:45.265722 18002 tablet_bootstrap.cc:492] T a600487619334a42810f4fdea0f11725 P fb7f98e62a7e4cfcabb04656fc6d03a0: Bootstrap starting.
I20260928 05:21:45.269392 18002 tablet_bootstrap.cc:654] T a600487619334a42810f4fdea0f11725 P fb7f98e62a7e4cfcabb04656fc6d03a0: Neither blocks nor log segments found. Creating new log.
I20260928 05:21:45.269482 18003 tablet_bootstrap.cc:492] T a600487619334a42810f4fdea0f11725 P e8b12dcaf16d4483aa41523ee70b768f: Bootstrap starting.
I20260928 05:21:45.270613 18002 log.cc:824] T a600487619334a42810f4fdea0f11725 P fb7f98e62a7e4cfcabb04656fc6d03a0: Log is configured to *not* fsync() on all Append() calls
I20260928 05:21:45.272693 18003 tablet_bootstrap.cc:654] T a600487619334a42810f4fdea0f11725 P e8b12dcaf16d4483aa41523ee70b768f: Neither blocks nor log segments found. Creating new log.
I20260928 05:21:45.272936 18002 tablet_bootstrap.cc:492] T a600487619334a42810f4fdea0f11725 P fb7f98e62a7e4cfcabb04656fc6d03a0: No bootstrap required, opened a new log
I20260928 05:21:45.273162 18002 ts_tablet_manager.cc:1397] T a600487619334a42810f4fdea0f11725 P fb7f98e62a7e4cfcabb04656fc6d03a0: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260928 05:21:45.273859 18003 log.cc:824] T a600487619334a42810f4fdea0f11725 P e8b12dcaf16d4483aa41523ee70b768f: Log is configured to *not* fsync() on all Append() calls
I20260928 05:21:45.274282 18005 tablet_bootstrap.cc:492] T a600487619334a42810f4fdea0f11725 P 0b36a61cea5c429a8c9621ccb934aa08: Bootstrap starting.
I20260928 05:21:45.277463 18005 tablet_bootstrap.cc:654] T a600487619334a42810f4fdea0f11725 P 0b36a61cea5c429a8c9621ccb934aa08: Neither blocks nor log segments found. Creating new log.
I20260928 05:21:45.278609 18005 log.cc:824] T a600487619334a42810f4fdea0f11725 P 0b36a61cea5c429a8c9621ccb934aa08: Log is configured to *not* fsync() on all Append() calls
I20260928 05:21:45.279016 18003 tablet_bootstrap.cc:492] T a600487619334a42810f4fdea0f11725 P e8b12dcaf16d4483aa41523ee70b768f: No bootstrap required, opened a new log
I20260928 05:21:45.279291 18003 ts_tablet_manager.cc:1397] T a600487619334a42810f4fdea0f11725 P e8b12dcaf16d4483aa41523ee70b768f: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.005s
I20260928 05:21:45.281363 18005 tablet_bootstrap.cc:492] T a600487619334a42810f4fdea0f11725 P 0b36a61cea5c429a8c9621ccb934aa08: No bootstrap required, opened a new log
I20260928 05:21:45.281597 18005 ts_tablet_manager.cc:1397] T a600487619334a42810f4fdea0f11725 P 0b36a61cea5c429a8c9621ccb934aa08: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.001s
I20260928 05:21:45.282919 18002 raft_consensus.cc:348] T a600487619334a42810f4fdea0f11725 P fb7f98e62a7e4cfcabb04656fc6d03a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb7f98e62a7e4cfcabb04656fc6d03a0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44193 } } peers { permanent_uuid: "0b36a61cea5c429a8c9621ccb934aa08" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43377 } } peers { permanent_uuid: "e8b12dcaf16d4483aa41523ee70b768f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38815 } }
I20260928 05:21:45.283390 18002 raft_consensus.cc:374] T a600487619334a42810f4fdea0f11725 P fb7f98e62a7e4cfcabb04656fc6d03a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:21:45.283499 18002 raft_consensus.cc:729] T a600487619334a42810f4fdea0f11725 P fb7f98e62a7e4cfcabb04656fc6d03a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb7f98e62a7e4cfcabb04656fc6d03a0, State: Initialized, Role: FOLLOWER
I20260928 05:21:45.284066 18002 consensus_queue.cc:260] T a600487619334a42810f4fdea0f11725 P fb7f98e62a7e4cfcabb04656fc6d03a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb7f98e62a7e4cfcabb04656fc6d03a0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44193 } } peers { permanent_uuid: "0b36a61cea5c429a8c9621ccb934aa08" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43377 } } peers { permanent_uuid: "e8b12dcaf16d4483aa41523ee70b768f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38815 } }
I20260928 05:21:45.286295 18003 raft_consensus.cc:348] T a600487619334a42810f4fdea0f11725 P e8b12dcaf16d4483aa41523ee70b768f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb7f98e62a7e4cfcabb04656fc6d03a0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44193 } } peers { permanent_uuid: "0b36a61cea5c429a8c9621ccb934aa08" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43377 } } peers { permanent_uuid: "e8b12dcaf16d4483aa41523ee70b768f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38815 } }
I20260928 05:21:45.286584 18003 raft_consensus.cc:374] T a600487619334a42810f4fdea0f11725 P e8b12dcaf16d4483aa41523ee70b768f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:21:45.286686 18003 raft_consensus.cc:729] T a600487619334a42810f4fdea0f11725 P e8b12dcaf16d4483aa41523ee70b768f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8b12dcaf16d4483aa41523ee70b768f, State: Initialized, Role: FOLLOWER
I20260928 05:21:45.287176 18003 consensus_queue.cc:260] T a600487619334a42810f4fdea0f11725 P e8b12dcaf16d4483aa41523ee70b768f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb7f98e62a7e4cfcabb04656fc6d03a0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44193 } } peers { permanent_uuid: "0b36a61cea5c429a8c9621ccb934aa08" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43377 } } peers { permanent_uuid: "e8b12dcaf16d4483aa41523ee70b768f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38815 } }
I20260928 05:21:45.288771 18003 ts_tablet_manager.cc:1428] T a600487619334a42810f4fdea0f11725 P e8b12dcaf16d4483aa41523ee70b768f: Time spent starting tablet: real 0.009s user 0.009s sys 0.000s
I20260928 05:21:45.289767 17984 heartbeater.cc:503] Master 127.0.97.62:45091 was elected leader, sending a full tablet report...
I20260928 05:21:45.290313 18002 ts_tablet_manager.cc:1428] T a600487619334a42810f4fdea0f11725 P fb7f98e62a7e4cfcabb04656fc6d03a0: Time spent starting tablet: real 0.017s user 0.011s sys 0.005s
I20260928 05:21:45.291579 18005 raft_consensus.cc:348] T a600487619334a42810f4fdea0f11725 P 0b36a61cea5c429a8c9621ccb934aa08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb7f98e62a7e4cfcabb04656fc6d03a0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44193 } } peers { permanent_uuid: "0b36a61cea5c429a8c9621ccb934aa08" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43377 } } peers { permanent_uuid: "e8b12dcaf16d4483aa41523ee70b768f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38815 } }
I20260928 05:21:45.292047 18005 raft_consensus.cc:374] T a600487619334a42810f4fdea0f11725 P 0b36a61cea5c429a8c9621ccb934aa08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:21:45.292220 18005 raft_consensus.cc:729] T a600487619334a42810f4fdea0f11725 P 0b36a61cea5c429a8c9621ccb934aa08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b36a61cea5c429a8c9621ccb934aa08, State: Initialized, Role: FOLLOWER
I20260928 05:21:45.292812 18005 consensus_queue.cc:260] T a600487619334a42810f4fdea0f11725 P 0b36a61cea5c429a8c9621ccb934aa08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb7f98e62a7e4cfcabb04656fc6d03a0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44193 } } peers { permanent_uuid: "0b36a61cea5c429a8c9621ccb934aa08" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43377 } } peers { permanent_uuid: "e8b12dcaf16d4483aa41523ee70b768f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38815 } }
I20260928 05:21:45.293846 17851 heartbeater.cc:503] Master 127.0.97.62:45091 was elected leader, sending a full tablet report...
I20260928 05:21:45.295316 18005 ts_tablet_manager.cc:1428] T a600487619334a42810f4fdea0f11725 P 0b36a61cea5c429a8c9621ccb934aa08: Time spent starting tablet: real 0.014s user 0.002s sys 0.010s
I20260928 05:21:45.361071 18008 raft_consensus.cc:482] T a600487619334a42810f4fdea0f11725 P fb7f98e62a7e4cfcabb04656fc6d03a0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:21:45.361334 18008 raft_consensus.cc:504] T a600487619334a42810f4fdea0f11725 P fb7f98e62a7e4cfcabb04656fc6d03a0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb7f98e62a7e4cfcabb04656fc6d03a0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44193 } } peers { permanent_uuid: "0b36a61cea5c429a8c9621ccb934aa08" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43377 } } peers { permanent_uuid: "e8b12dcaf16d4483aa41523ee70b768f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38815 } }
I20260928 05:21:45.362890 18008 leader_election.cc:288] T a600487619334a42810f4fdea0f11725 P fb7f98e62a7e4cfcabb04656fc6d03a0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b36a61cea5c429a8c9621ccb934aa08 (127.0.97.2:43377), e8b12dcaf16d4483aa41523ee70b768f (127.0.97.1:38815)
I20260928 05:21:45.368795 17999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:45.363201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35693 (local address 127.0.97.2:43377)
0928 05:21:45.363394 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 05:21:45.363400 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:21:45.363420 (+ 20us) server_negotiation.cc:407] Connection header received
0928 05:21:45.363620 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:21:45.363625 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:21:45.363700 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:21:45.363781 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:21:45.364232 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:45.365200 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:45.366958 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:45.368038 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:45.368132 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:45.368197 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 05:21:45.368527 (+ 330us) server_negotiation.cc:299] Negotiation successful
0928 05:21:45.368610 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 05:21:45.368909 17986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:45.363092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43377 (local address 127.0.97.3:35693)
0928 05:21:45.363338 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:45.363359 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 05:21:45.363489 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:21:45.363868 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:21:45.363874 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:21:45.363900 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:21:45.364090 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:45.364098 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:45.365325 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:45.365330 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:45.366815 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:45.366827 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:45.368173 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:45.368179 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:45.368368 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:45.368404 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 05:21:45.368504 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 05:21:45.368630 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96}
I20260928 05:21:45.369756 18011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:45.363448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38815 (local address 127.0.97.3:38827)
0928 05:21:45.364083 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:45.364097 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 05:21:45.364207 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:21:45.364476 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:21:45.364480 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:21:45.364514 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:21:45.364813 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:45.364823 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:45.366010 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:45.366014 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:45.367625 (+ 1611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:45.367639 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:45.369164 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:45.369168 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:45.369322 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:45.369351 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 05:21:45.369441 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 05:21:45.369538 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":339,"threads_started":1}
I20260928 05:21:45.369824 18001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:45.363577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38827 (local address 127.0.97.1:38815)
0928 05:21:45.363774 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 05:21:45.363780 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:21:45.364182 (+ 402us) server_negotiation.cc:407] Connection header received
0928 05:21:45.364322 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:21:45.364326 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:21:45.364382 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:21:45.364455 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:21:45.364973 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:45.365873 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:45.367783 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:45.368818 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:45.368966 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:45.369064 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 05:21:45.369489 (+ 425us) server_negotiation.cc:299] Negotiation successful
0928 05:21:45.369611 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 05:21:45.370011 17806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a600487619334a42810f4fdea0f11725" candidate_uuid: "fb7f98e62a7e4cfcabb04656fc6d03a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b36a61cea5c429a8c9621ccb934aa08" is_pre_election: true
I20260928 05:21:45.370512 17806 raft_consensus.cc:2454] T a600487619334a42810f4fdea0f11725 P 0b36a61cea5c429a8c9621ccb934aa08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb7f98e62a7e4cfcabb04656fc6d03a0 in term 0.
I20260928 05:21:45.370643 17672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a600487619334a42810f4fdea0f11725" candidate_uuid: "fb7f98e62a7e4cfcabb04656fc6d03a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8b12dcaf16d4483aa41523ee70b768f" is_pre_election: true
I20260928 05:21:45.371096 17875 leader_election.cc:302] T a600487619334a42810f4fdea0f11725 P fb7f98e62a7e4cfcabb04656fc6d03a0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b36a61cea5c429a8c9621ccb934aa08, fb7f98e62a7e4cfcabb04656fc6d03a0; no voters:
I20260928 05:21:45.371119 17672 raft_consensus.cc:2454] T a600487619334a42810f4fdea0f11725 P e8b12dcaf16d4483aa41523ee70b768f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb7f98e62a7e4cfcabb04656fc6d03a0 in term 0.
I20260928 05:21:45.371416 18008 raft_consensus.cc:2781] T a600487619334a42810f4fdea0f11725 P fb7f98e62a7e4cfcabb04656fc6d03a0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 05:21:45.371554 18008 raft_consensus.cc:482] T a600487619334a42810f4fdea0f11725 P fb7f98e62a7e4cfcabb04656fc6d03a0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 05:21:45.371668 18008 raft_consensus.cc:3037] T a600487619334a42810f4fdea0f11725 P fb7f98e62a7e4cfcabb04656fc6d03a0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:21:45.373814 18008 raft_consensus.cc:504] T a600487619334a42810f4fdea0f11725 P fb7f98e62a7e4cfcabb04656fc6d03a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb7f98e62a7e4cfcabb04656fc6d03a0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44193 } } peers { permanent_uuid: "0b36a61cea5c429a8c9621ccb934aa08" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43377 } } peers { permanent_uuid: "e8b12dcaf16d4483aa41523ee70b768f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38815 } }
I20260928 05:21:45.374353 18008 leader_election.cc:288] T a600487619334a42810f4fdea0f11725 P fb7f98e62a7e4cfcabb04656fc6d03a0 [CANDIDATE]: Term 1 election: Requested vote from peers 0b36a61cea5c429a8c9621ccb934aa08 (127.0.97.2:43377), e8b12dcaf16d4483aa41523ee70b768f (127.0.97.1:38815)
I20260928 05:21:45.374735 17806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a600487619334a42810f4fdea0f11725" candidate_uuid: "fb7f98e62a7e4cfcabb04656fc6d03a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b36a61cea5c429a8c9621ccb934aa08"
I20260928 05:21:45.374801 17672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a600487619334a42810f4fdea0f11725" candidate_uuid: "fb7f98e62a7e4cfcabb04656fc6d03a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8b12dcaf16d4483aa41523ee70b768f"
I20260928 05:21:45.374887 17806 raft_consensus.cc:3037] T a600487619334a42810f4fdea0f11725 P 0b36a61cea5c429a8c9621ccb934aa08 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:21:45.374982 17672 raft_consensus.cc:3037] T a600487619334a42810f4fdea0f11725 P e8b12dcaf16d4483aa41523ee70b768f [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:21:45.377022 17806 raft_consensus.cc:2454] T a600487619334a42810f4fdea0f11725 P 0b36a61cea5c429a8c9621ccb934aa08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb7f98e62a7e4cfcabb04656fc6d03a0 in term 1.
I20260928 05:21:45.377069 17672 raft_consensus.cc:2454] T a600487619334a42810f4fdea0f11725 P e8b12dcaf16d4483aa41523ee70b768f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb7f98e62a7e4cfcabb04656fc6d03a0 in term 1.
I20260928 05:21:45.377347 17875 leader_election.cc:302] T a600487619334a42810f4fdea0f11725 P fb7f98e62a7e4cfcabb04656fc6d03a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b36a61cea5c429a8c9621ccb934aa08, fb7f98e62a7e4cfcabb04656fc6d03a0; no voters:
I20260928 05:21:45.377615 18008 raft_consensus.cc:2781] T a600487619334a42810f4fdea0f11725 P fb7f98e62a7e4cfcabb04656fc6d03a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:21:45.378024 18008 raft_consensus.cc:686] T a600487619334a42810f4fdea0f11725 P fb7f98e62a7e4cfcabb04656fc6d03a0 [term 1 LEADER]: Becoming Leader. State: Replica: fb7f98e62a7e4cfcabb04656fc6d03a0, State: Running, Role: LEADER
I20260928 05:21:45.378443 18008 consensus_queue.cc:237] T a600487619334a42810f4fdea0f11725 P fb7f98e62a7e4cfcabb04656fc6d03a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb7f98e62a7e4cfcabb04656fc6d03a0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44193 } } peers { permanent_uuid: "0b36a61cea5c429a8c9621ccb934aa08" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43377 } } peers { permanent_uuid: "e8b12dcaf16d4483aa41523ee70b768f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38815 } }
I20260928 05:21:45.382802 17528 catalog_manager.cc:5495] T a600487619334a42810f4fdea0f11725 P fb7f98e62a7e4cfcabb04656fc6d03a0 reported cstate change: term changed from 0 to 1, leader changed from <none> to fb7f98e62a7e4cfcabb04656fc6d03a0 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "fb7f98e62a7e4cfcabb04656fc6d03a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb7f98e62a7e4cfcabb04656fc6d03a0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b36a61cea5c429a8c9621ccb934aa08" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8b12dcaf16d4483aa41523ee70b768f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38815 } health_report { overall_health: UNKNOWN } } }
W20260928 05:21:45.406047 17985 tablet.cc:2369] T a600487619334a42810f4fdea0f11725 P fb7f98e62a7e4cfcabb04656fc6d03a0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 05:21:45.428786 17998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:45.423464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57636 (local address 127.0.97.3:44193)
0928 05:21:45.423625 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 05:21:45.423630 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:21:45.423648 (+ 18us) server_negotiation.cc:407] Connection header received
0928 05:21:45.423810 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:21:45.423813 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:21:45.423893 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:21:45.424018 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:21:45.424311 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:45.425034 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:45.426483 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:45.426986 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:45.427113 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:45.427619 (+ 506us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:21:45.428092 (+ 473us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:21:45.428217 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 05:21:45.428411 (+ 194us) server_negotiation.cc:299] Negotiation successful
0928 05:21:45.428504 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 05:21:45.436147 17672 raft_consensus.cc:1261] T a600487619334a42810f4fdea0f11725 P e8b12dcaf16d4483aa41523ee70b768f [term 1 FOLLOWER]: Refusing update from remote peer fb7f98e62a7e4cfcabb04656fc6d03a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:21:45.436177 17806 raft_consensus.cc:1261] T a600487619334a42810f4fdea0f11725 P 0b36a61cea5c429a8c9621ccb934aa08 [term 1 FOLLOWER]: Refusing update from remote peer fb7f98e62a7e4cfcabb04656fc6d03a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:21:45.436985 18012 consensus_queue.cc:1035] T a600487619334a42810f4fdea0f11725 P fb7f98e62a7e4cfcabb04656fc6d03a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b36a61cea5c429a8c9621ccb934aa08" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:21:45.437323 18008 consensus_queue.cc:1035] T a600487619334a42810f4fdea0f11725 P fb7f98e62a7e4cfcabb04656fc6d03a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8b12dcaf16d4483aa41523ee70b768f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 05:21:45.446839 17852 tablet.cc:2369] T a600487619334a42810f4fdea0f11725 P 0b36a61cea5c429a8c9621ccb934aa08: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 05:21:45.449110 18016 mvcc.cc:205] Tried to move back new op lower bound from 7334186620657561600 to 7334186620433666048. Current Snapshot: MvccSnapshot[applied={T|T < 7334186620657561600}]
I20260928 05:21:45.460041 17526 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54654:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:21:45.460287 17526 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54654:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:21:45.463591 17526 catalog_manager.cc:5782] T 00000000000000000000000000000000 P de8ad57463bc4b5aba2224dfb24bd193: Sending DeleteTablet for 3 replicas of tablet a600487619334a42810f4fdea0f11725
I20260928 05:21:45.464548 17919 tablet_service.cc:1514] Processing DeleteTablet for tablet a600487619334a42810f4fdea0f11725 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:21:45 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57628
I20260928 05:21:45.464661 17786 tablet_service.cc:1514] Processing DeleteTablet for tablet a600487619334a42810f4fdea0f11725 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:21:45 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56946
I20260928 05:21:45.465197 17652 tablet_service.cc:1514] Processing DeleteTablet for tablet a600487619334a42810f4fdea0f11725 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:21:45 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49126
I20260928 05:21:45.467047 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 17589
I20260928 05:21:45.475242 18027 tablet_replica.cc:330] T a600487619334a42810f4fdea0f11725 P e8b12dcaf16d4483aa41523ee70b768f: stopping tablet replica
I20260928 05:21:45.476221 18027 raft_consensus.cc:2229] T a600487619334a42810f4fdea0f11725 P e8b12dcaf16d4483aa41523ee70b768f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:21:45.476893 18026 tablet_replica.cc:330] T a600487619334a42810f4fdea0f11725 P 0b36a61cea5c429a8c9621ccb934aa08: stopping tablet replica
I20260928 05:21:45.477294 18027 raft_consensus.cc:2258] T a600487619334a42810f4fdea0f11725 P e8b12dcaf16d4483aa41523ee70b768f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:21:45.477403 18026 raft_consensus.cc:2229] T a600487619334a42810f4fdea0f11725 P 0b36a61cea5c429a8c9621ccb934aa08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:21:45.477880 18026 raft_consensus.cc:2258] T a600487619334a42810f4fdea0f11725 P 0b36a61cea5c429a8c9621ccb934aa08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:21:45.478636 18025 tablet_replica.cc:330] T a600487619334a42810f4fdea0f11725 P fb7f98e62a7e4cfcabb04656fc6d03a0: stopping tablet replica
I20260928 05:21:45.480281 18026 ts_tablet_manager.cc:1905] T a600487619334a42810f4fdea0f11725 P 0b36a61cea5c429a8c9621ccb934aa08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:21:45.480285 18027 ts_tablet_manager.cc:1905] T a600487619334a42810f4fdea0f11725 P e8b12dcaf16d4483aa41523ee70b768f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:21:45.483794 18025 raft_consensus.cc:2229] T a600487619334a42810f4fdea0f11725 P fb7f98e62a7e4cfcabb04656fc6d03a0 [term 1 LEADER]: Raft consensus shutting down.
I20260928 05:21:45.484721 18025 raft_consensus.cc:2258] T a600487619334a42810f4fdea0f11725 P fb7f98e62a7e4cfcabb04656fc6d03a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:21:45.486883 18025 ts_tablet_manager.cc:1905] T a600487619334a42810f4fdea0f11725 P fb7f98e62a7e4cfcabb04656fc6d03a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:21:45.486982 18027 ts_tablet_manager.cc:1918] T a600487619334a42810f4fdea0f11725 P e8b12dcaf16d4483aa41523ee70b768f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:21:45.487175 18027 log.cc:1196] T a600487619334a42810f4fdea0f11725 P e8b12dcaf16d4483aa41523ee70b768f: Deleting WAL directory at /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572622576160-388-0/minicluster-data/ts-0/wal/wals/a600487619334a42810f4fdea0f11725
I20260928 05:21:45.487715 18027 ts_tablet_manager.cc:1939] T a600487619334a42810f4fdea0f11725 P e8b12dcaf16d4483aa41523ee70b768f: Deleting consensus metadata
I20260928 05:21:45.489430 17515 catalog_manager.cc:4841] TS e8b12dcaf16d4483aa41523ee70b768f (127.0.97.1:38815): tablet a600487619334a42810f4fdea0f11725 (table test-table [id=8218d2ff4c9a4944a92f8db5bb95df94]) successfully deleted
Sep 28 05:21:45 dist-test-slave-whsn krb5kdc[17483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572903, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260928 05:21:45.490996 18026 ts_tablet_manager.cc:1918] T a600487619334a42810f4fdea0f11725 P 0b36a61cea5c429a8c9621ccb934aa08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:21:45.491146 18026 log.cc:1196] T a600487619334a42810f4fdea0f11725 P 0b36a61cea5c429a8c9621ccb934aa08: Deleting WAL directory at /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572622576160-388-0/minicluster-data/ts-1/wal/wals/a600487619334a42810f4fdea0f11725
I20260928 05:21:45.491627 18026 ts_tablet_manager.cc:1939] T a600487619334a42810f4fdea0f11725 P 0b36a61cea5c429a8c9621ccb934aa08: Deleting consensus metadata
I20260928 05:21:45.493530 17515 catalog_manager.cc:4841] TS 0b36a61cea5c429a8c9621ccb934aa08 (127.0.97.2:43377): tablet a600487619334a42810f4fdea0f11725 (table test-table [id=8218d2ff4c9a4944a92f8db5bb95df94]) successfully deleted
I20260928 05:21:45.497181 18001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:45.468467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49134 (local address 127.0.97.1:38815)
0928 05:21:45.468639 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 05:21:45.468646 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:21:45.481254 (+ 12608us) server_negotiation.cc:407] Connection header received
0928 05:21:45.481643 (+ 389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:21:45.481648 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:21:45.481731 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:21:45.481803 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:21:45.483579 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:45.484687 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:45.486539 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:45.487172 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:45.487266 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:45.491841 (+ 4575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:21:45.491851 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:21:45.491863 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:21:45.491891 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:21:45.494616 (+ 2725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:21:45.495212 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:21:45.495218 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:21:45.495220 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:21:45.495331 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:21:45.495677 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:21:45.495681 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:21:45.495683 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:21:45.496001 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:21:45.496269 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0928 05:21:45.496564 (+ 295us) server_negotiation.cc:299] Negotiation successful
0928 05:21:45.496754 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 05:21:45.497756 17712 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:21:45.499922 18025 ts_tablet_manager.cc:1918] T a600487619334a42810f4fdea0f11725 P fb7f98e62a7e4cfcabb04656fc6d03a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:21:45.500169 18025 log.cc:1196] T a600487619334a42810f4fdea0f11725 P fb7f98e62a7e4cfcabb04656fc6d03a0: Deleting WAL directory at /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572622576160-388-0/minicluster-data/ts-2/wal/wals/a600487619334a42810f4fdea0f11725
I20260928 05:21:45.500715 18025 ts_tablet_manager.cc:1939] T a600487619334a42810f4fdea0f11725 P fb7f98e62a7e4cfcabb04656fc6d03a0: Deleting consensus metadata
I20260928 05:21:45.502362 17513 catalog_manager.cc:4841] TS fb7f98e62a7e4cfcabb04656fc6d03a0 (127.0.97.3:44193): tablet a600487619334a42810f4fdea0f11725 (table test-table [id=8218d2ff4c9a4944a92f8db5bb95df94]) successfully deleted
I20260928 05:21:45.583611 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 17589
I20260928 05:21:45.598140 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 17723
Sep 28 05:21:45 dist-test-slave-whsn krb5kdc[17483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572903, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260928 05:21:45.612746 17999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:45.598925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56950 (local address 127.0.97.2:43377)
0928 05:21:45.599117 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 05:21:45.599123 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:21:45.599142 (+ 19us) server_negotiation.cc:407] Connection header received
0928 05:21:45.599201 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:21:45.599207 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:21:45.599271 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:21:45.599372 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:21:45.600653 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:45.601521 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:45.602758 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:45.603346 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:45.603462 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:45.607554 (+ 4092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:21:45.607568 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:21:45.607584 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:21:45.607618 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:21:45.610267 (+ 2649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:21:45.610877 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:21:45.610887 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:21:45.610891 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:21:45.611017 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:21:45.611329 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:21:45.611334 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:21:45.611338 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:21:45.611685 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:21:45.611954 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0928 05:21:45.612238 (+ 284us) server_negotiation.cc:299] Negotiation successful
0928 05:21:45.612437 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 05:21:45.613421 17846 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:21:45.702571 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 17723
I20260928 05:21:45.722900 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 17856
Sep 28 05:21:45 dist-test-slave-whsn krb5kdc[17483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572903, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260928 05:21:45.739283 17998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:45.723583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57638 (local address 127.0.97.3:44193)
0928 05:21:45.723725 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 05:21:45.723729 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:21:45.723787 (+ 58us) server_negotiation.cc:407] Connection header received
0928 05:21:45.723960 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:21:45.723964 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:21:45.724035 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:21:45.724117 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:21:45.725932 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:45.726751 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:45.728545 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:45.729032 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:45.729117 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:45.734015 (+ 4898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:21:45.734027 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:21:45.734042 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:21:45.734079 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:21:45.736742 (+ 2663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:21:45.737359 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:21:45.737366 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:21:45.737368 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:21:45.737516 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:21:45.737861 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:21:45.737864 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:21:45.737866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:21:45.738204 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:21:45.738500 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0928 05:21:45.738751 (+ 251us) server_negotiation.cc:299] Negotiation successful
0928 05:21:45.738945 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 05:21:45.739976 17979 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:21:45.856006 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 17856
I20260928 05:21:45.871244 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 17496
I20260928 05:21:45.872318 17558 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:21:45.979569 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 17496
2026-09-28T05:21:45Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (3907 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-28T05:21:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:21:46Z Disabled control of system clock
I20260928 05:21:46.043753 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572622576160-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572622576160-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572622576160-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33759
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:38021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33759
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 05:21:46.349623 18036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:21:46.349887 18036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:21:46.349963 18036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:21:46.356788 18036 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:21:46.356897 18036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:21:46.356943 18036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:21:46.356997 18036 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:21:46.357040 18036 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:21:46.365690 18036 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572622576160-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572622576160-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33759
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33759
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572622576160-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:21:46.366819 18036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:21:46.368391 18036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:21:46.379065 18042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:21:46.379102 18044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:21:46.379125 18041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:21:46.379793 18036 server_base.cc:1034] running on GCE node
I20260928 05:21:46.380456 18036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:21:46.381798 18036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:21:46.382983 18036 hybrid_clock.cc:648] HybridClock initialized: now 1790572906382936 us; error 44 us; skew 500 ppm
I20260928 05:21:46.383280 18036 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 05:21:46.388622 18036 webserver.cc:460] Webserver started at http://127.0.97.62:41915/ using document root <none> and password file <none>
I20260928 05:21:46.389303 18036 fs_manager.cc:359] Metadata directory not provided
I20260928 05:21:46.389400 18036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:21:46.389693 18036 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:21:46.392445 18036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572622576160-388-0/minicluster-data/master-0/data/instance:
uuid: "7add23e8d8b649bdbeac9df8e437a966"
format_stamp: "Formatted at 2026-09-28 05:21:46 on dist-test-slave-whsn"
server_key: "aa0d49f563137289e2bd8c2bf723d56d"
server_key_iv: "55309642a6ae39d1c27f6f8c8c56cfa1"
server_key_version: "encryptionkey@0"
I20260928 05:21:46.393210 18036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572622576160-388-0/minicluster-data/master-0/wal/instance:
uuid: "7add23e8d8b649bdbeac9df8e437a966"
format_stamp: "Formatted at 2026-09-28 05:21:46 on dist-test-slave-whsn"
server_key: "aa0d49f563137289e2bd8c2bf723d56d"
server_key_iv: "55309642a6ae39d1c27f6f8c8c56cfa1"
server_key_version: "encryptionkey@0"
I20260928 05:21:46.397810 18036 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 05:21:46.400895 18050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:21:46.402115 18036 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:21:46.402268 18036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572622576160-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572622576160-388-0/minicluster-data/master-0/wal
uuid: "7add23e8d8b649bdbeac9df8e437a966"
format_stamp: "Formatted at 2026-09-28 05:21:46 on dist-test-slave-whsn"
server_key: "aa0d49f563137289e2bd8c2bf723d56d"
server_key_iv: "55309642a6ae39d1c27f6f8c8c56cfa1"
server_key_version: "encryptionkey@0"
I20260928 05:21:46.402450 18036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572622576160-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572622576160-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572622576160-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:21:46.416239 18036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:21:46.416963 18036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:21:46.417199 18036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:21:46.433429 18036 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33759
I20260928 05:21:46.433459 18101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33759 every 8 connection(s)
I20260928 05:21:46.434602 18036 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:21:46.437932 18102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:21:46.441969 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 18036
I20260928 05:21:46.442131 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572622576160-388-0/minicluster-data/master-0/wal/instance
I20260928 05:21:46.442623 388 external_mini_cluster.cc:1428] Setting key 802763df493958a3c897a601dd09ff47
I20260928 05:21:46.448305 18102 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7add23e8d8b649bdbeac9df8e437a966: Bootstrap starting.
I20260928 05:21:46.451589 18102 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7add23e8d8b649bdbeac9df8e437a966: Neither blocks nor log segments found. Creating new log.
I20260928 05:21:46.452662 18102 log.cc:824] T 00000000000000000000000000000000 P 7add23e8d8b649bdbeac9df8e437a966: Log is configured to *not* fsync() on all Append() calls
I20260928 05:21:46.455405 18102 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7add23e8d8b649bdbeac9df8e437a966: No bootstrap required, opened a new log
I20260928 05:21:46.455610 18105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:46.444301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52094 (local address 127.0.97.62:33759)
0928 05:21:46.445177 (+ 876us) server_negotiation.cc:206] Beginning negotiation
0928 05:21:46.445187 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 05:21:46.445225 (+ 38us) server_negotiation.cc:407] Connection header received
0928 05:21:46.446957 (+ 1732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:21:46.446977 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:21:46.447376 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:21:46.447668 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:21:46.448247 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:46.449595 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:46.451468 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:46.452136 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:46.452253 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:46.453465 (+ 1212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:21:46.453475 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:21:46.453492 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:21:46.454092 (+ 600us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:21:46.454161 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:21:46.454180 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:21:46.454258 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 05:21:46.454468 (+ 210us) server_negotiation.cc:299] Negotiation successful
0928 05:21:46.454747 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":477,"thread_start_us":289,"threads_started":1}
I20260928 05:21:46.462226 18102 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7add23e8d8b649bdbeac9df8e437a966 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7add23e8d8b649bdbeac9df8e437a966" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33759 } }
I20260928 05:21:46.462536 18102 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7add23e8d8b649bdbeac9df8e437a966 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:21:46.462618 18102 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7add23e8d8b649bdbeac9df8e437a966 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7add23e8d8b649bdbeac9df8e437a966, State: Initialized, Role: FOLLOWER
I20260928 05:21:46.463088 18102 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7add23e8d8b649bdbeac9df8e437a966 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7add23e8d8b649bdbeac9df8e437a966" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33759 } }
I20260928 05:21:46.463306 18102 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7add23e8d8b649bdbeac9df8e437a966 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:21:46.463419 18102 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7add23e8d8b649bdbeac9df8e437a966 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:21:46.463568 18102 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7add23e8d8b649bdbeac9df8e437a966 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:21:46.465632 18102 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7add23e8d8b649bdbeac9df8e437a966 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7add23e8d8b649bdbeac9df8e437a966" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33759 } }
I20260928 05:21:46.466045 18102 leader_election.cc:302] T 00000000000000000000000000000000 P 7add23e8d8b649bdbeac9df8e437a966 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7add23e8d8b649bdbeac9df8e437a966; no voters:
I20260928 05:21:46.466583 18102 leader_election.cc:288] T 00000000000000000000000000000000 P 7add23e8d8b649bdbeac9df8e437a966 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:21:46.466702 18107 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7add23e8d8b649bdbeac9df8e437a966 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:21:46.467391 18107 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7add23e8d8b649bdbeac9df8e437a966 [term 1 LEADER]: Becoming Leader. State: Replica: 7add23e8d8b649bdbeac9df8e437a966, State: Running, Role: LEADER
I20260928 05:21:46.467904 18107 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7add23e8d8b649bdbeac9df8e437a966 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7add23e8d8b649bdbeac9df8e437a966" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33759 } }
I20260928 05:21:46.468631 18102 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7add23e8d8b649bdbeac9df8e437a966 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:21:46.470547 18109 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7add23e8d8b649bdbeac9df8e437a966 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7add23e8d8b649bdbeac9df8e437a966. Latest consensus state: current_term: 1 leader_uuid: "7add23e8d8b649bdbeac9df8e437a966" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7add23e8d8b649bdbeac9df8e437a966" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33759 } } }
I20260928 05:21:46.470903 18109 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7add23e8d8b649bdbeac9df8e437a966 [sys.catalog]: This master's current role is: LEADER
I20260928 05:21:46.470846 18108 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7add23e8d8b649bdbeac9df8e437a966 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7add23e8d8b649bdbeac9df8e437a966" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7add23e8d8b649bdbeac9df8e437a966" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33759 } } }
I20260928 05:21:46.471211 18108 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7add23e8d8b649bdbeac9df8e437a966 [sys.catalog]: This master's current role is: LEADER
I20260928 05:21:46.472800 18115 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:21:46.477421 18115 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:21:46.485347 18115 catalog_manager.cc:1347] Generated new cluster ID: 5c497b2b7ed945d3853a9362f5bc66a2
I20260928 05:21:46.485440 18115 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:21:46.506502 18115 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:21:46.507710 18115 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:21:46.518188 18115 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7add23e8d8b649bdbeac9df8e437a966: Generated new TSK 0
I20260928 05:21:46.518893 18115 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 05:21:46.527976 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572622576160-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572622576160-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572622576160-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572622576160-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33759
--builtin_ntp_servers=127.0.97.20:38021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 05:21:46.837495 18126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:21:46.837745 18126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:21:46.837826 18126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:21:46.844614 18126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:21:46.844694 18126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:21:46.844797 18126 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 05:21:46.853312 18126 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572622576160-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572622576160-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572622576160-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33759
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572622576160-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:21:46.854353 18126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:21:46.855945 18126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:21:46.867631 18134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:21:46.867666 18132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:21:46.867705 18131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:21:46.868470 18126 server_base.cc:1034] running on GCE node
I20260928 05:21:46.869077 18126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:21:46.869908 18126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:21:46.871096 18126 hybrid_clock.cc:648] HybridClock initialized: now 1790572906871037 us; error 54 us; skew 500 ppm
I20260928 05:21:46.871383 18126 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 05:21:46.873423 18126 webserver.cc:460] Webserver started at http://127.0.97.1:41573/ using document root <none> and password file <none>
I20260928 05:21:46.874030 18126 fs_manager.cc:359] Metadata directory not provided
I20260928 05:21:46.874123 18126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:21:46.874405 18126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:21:46.876998 18126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572622576160-388-0/minicluster-data/ts-0/data/instance:
uuid: "4c4e7bf2fb584a6a898c1efc57215377"
format_stamp: "Formatted at 2026-09-28 05:21:46 on dist-test-slave-whsn"
server_key: "fd45fa13faae62ec9acdb9a316dc6ae2"
server_key_iv: "7d8783c756005deb52379cfe771a4a29"
server_key_version: "encryptionkey@0"
I20260928 05:21:46.877694 18126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572622576160-388-0/minicluster-data/ts-0/wal/instance:
uuid: "4c4e7bf2fb584a6a898c1efc57215377"
format_stamp: "Formatted at 2026-09-28 05:21:46 on dist-test-slave-whsn"
server_key: "fd45fa13faae62ec9acdb9a316dc6ae2"
server_key_iv: "7d8783c756005deb52379cfe771a4a29"
server_key_version: "encryptionkey@0"
I20260928 05:21:46.882452 18126 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 05:21:46.885592 18140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:21:46.886812 18126 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:21:46.886951 18126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572622576160-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572622576160-388-0/minicluster-data/ts-0/wal
uuid: "4c4e7bf2fb584a6a898c1efc57215377"
format_stamp: "Formatted at 2026-09-28 05:21:46 on dist-test-slave-whsn"
server_key: "fd45fa13faae62ec9acdb9a316dc6ae2"
server_key_iv: "7d8783c756005deb52379cfe771a4a29"
server_key_version: "encryptionkey@0"
I20260928 05:21:46.887137 18126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572622576160-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572622576160-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572622576160-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:21:46.932945 18126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:21:46.933732 18126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:21:46.933969 18126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:21:46.934885 18126 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:21:46.936259 18126 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:21:46.936357 18126 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:21:46.936455 18126 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:21:46.936528 18126 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:21:46.965075 18126 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43321
I20260928 05:21:46.965126 18252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43321 every 8 connection(s)
I20260928 05:21:46.966374 18126 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572622576160-388-0/minicluster-data/ts-0/data/info.pb
I20260928 05:21:46.973872 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 18126
I20260928 05:21:46.974028 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572622576160-388-0/minicluster-data/ts-0/wal/instance
I20260928 05:21:46.974520 388 external_mini_cluster.cc:1428] Setting key d76fd039d08448c6b0e793893cf640c8
I20260928 05:21:46.977321 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572622576160-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572622576160-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572622576160-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572622576160-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33759
--builtin_ntp_servers=127.0.97.20:38021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 05:21:46.978765 18255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:46.968696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53857 (local address 127.0.97.62:33759)
0928 05:21:46.969148 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0928 05:21:46.969152 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:21:46.970233 (+ 1081us) server_negotiation.cc:407] Connection header received
0928 05:21:46.972035 (+ 1802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:21:46.972040 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:21:46.972106 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:21:46.972214 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:21:46.973286 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:46.974225 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:46.976089 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:46.976624 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:46.976749 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:46.977999 (+ 1250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:21:46.978005 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:21:46.978018 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:21:46.978039 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:21:46.978087 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:21:46.978102 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:21:46.978188 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 05:21:46.978358 (+ 170us) server_negotiation.cc:299] Negotiation successful
0928 05:21:46.978447 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":227,"threads_started":1}
I20260928 05:21:46.979688 18256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:46.969199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33759 (local address 127.0.97.1:53857)
0928 05:21:46.970089 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:46.970130 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 05:21:46.971774 (+ 1644us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:21:46.972406 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:21:46.972414 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:21:46.972497 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:21:46.973134 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:46.973152 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:46.974364 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:46.974368 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:46.975935 (+ 1567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:46.975948 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:46.976818 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:46.976823 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:46.977064 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:46.977711 (+ 647us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 05:21:46.977714 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:21:46.977751 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 05:21:46.977825 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:21:46.978184 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:21:46.978190 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:21:46.978195 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 05:21:46.978319 (+ 124us) client_negotiation.cc:240] Negotiation successful
0928 05:21:46.978608 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":266,"threads_started":1}
I20260928 05:21:46.980994 18253 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33759
I20260928 05:21:46.981284 18253 heartbeater.cc:464] Registering TS with master...
I20260928 05:21:46.982111 18253 heartbeater.cc:511] Master 127.0.97.62:33759 requested a full tablet report, sending...
I20260928 05:21:46.984056 18067 ts_manager.cc:194] Registered new tserver with Master: 4c4e7bf2fb584a6a898c1efc57215377 (127.0.97.1:43321)
I20260928 05:21:46.985503 18067 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:53857
W20260928 05:21:47.282166 18257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:21:47.282425 18257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:21:47.282553 18257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:21:47.289561 18257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:21:47.289659 18257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:21:47.289794 18257 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 05:21:47.298480 18257 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572622576160-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572622576160-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572622576160-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33759
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572622576160-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:21:47.299624 18257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:21:47.301290 18257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:21:47.312907 18262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:21:47.313007 18265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:21:47.312986 18263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:21:47.313930 18257 server_base.cc:1034] running on GCE node
I20260928 05:21:47.314464 18257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:21:47.315343 18257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:21:47.316542 18257 hybrid_clock.cc:648] HybridClock initialized: now 1790572907316481 us; error 44 us; skew 500 ppm
I20260928 05:21:47.316815 18257 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 05:21:47.318786 18257 webserver.cc:460] Webserver started at http://127.0.97.2:45435/ using document root <none> and password file <none>
I20260928 05:21:47.319387 18257 fs_manager.cc:359] Metadata directory not provided
I20260928 05:21:47.319487 18257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:21:47.319763 18257 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:21:47.322389 18257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572622576160-388-0/minicluster-data/ts-1/data/instance:
uuid: "d04f564474294351a9377ed80ae56d69"
format_stamp: "Formatted at 2026-09-28 05:21:47 on dist-test-slave-whsn"
server_key: "201ad607064a0825d63d15cc126b25d8"
server_key_iv: "30bbda74c19bf3e688217ea88faa1f2a"
server_key_version: "encryptionkey@0"
I20260928 05:21:47.323093 18257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572622576160-388-0/minicluster-data/ts-1/wal/instance:
uuid: "d04f564474294351a9377ed80ae56d69"
format_stamp: "Formatted at 2026-09-28 05:21:47 on dist-test-slave-whsn"
server_key: "201ad607064a0825d63d15cc126b25d8"
server_key_iv: "30bbda74c19bf3e688217ea88faa1f2a"
server_key_version: "encryptionkey@0"
I20260928 05:21:47.327579 18257 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260928 05:21:47.330658 18271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:21:47.331849 18257 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 05:21:47.331990 18257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572622576160-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572622576160-388-0/minicluster-data/ts-1/wal
uuid: "d04f564474294351a9377ed80ae56d69"
format_stamp: "Formatted at 2026-09-28 05:21:47 on dist-test-slave-whsn"
server_key: "201ad607064a0825d63d15cc126b25d8"
server_key_iv: "30bbda74c19bf3e688217ea88faa1f2a"
server_key_version: "encryptionkey@0"
I20260928 05:21:47.332166 18257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572622576160-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572622576160-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572622576160-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:21:47.356998 18257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:21:47.357770 18257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:21:47.358002 18257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:21:47.358877 18257 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:21:47.360183 18257 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:21:47.360271 18257 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:21:47.360368 18257 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:21:47.360437 18257 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:21:47.388908 18257 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:39289
I20260928 05:21:47.388959 18383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:39289 every 8 connection(s)
I20260928 05:21:47.390197 18257 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572622576160-388-0/minicluster-data/ts-1/data/info.pb
I20260928 05:21:47.399667 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 18257
I20260928 05:21:47.399834 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572622576160-388-0/minicluster-data/ts-1/wal/instance
I20260928 05:21:47.400372 388 external_mini_cluster.cc:1428] Setting key 0a30fc2d2c60220ffc173fe638410ff2
I20260928 05:21:47.401669 18255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:47.392554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48489 (local address 127.0.97.62:33759)
0928 05:21:47.392708 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 05:21:47.392712 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:21:47.394057 (+ 1345us) server_negotiation.cc:407] Connection header received
0928 05:21:47.395694 (+ 1637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:21:47.395698 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:21:47.395766 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:21:47.395849 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:21:47.396790 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:47.397516 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:47.398916 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:47.399286 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:47.399367 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:47.400751 (+ 1384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:21:47.400755 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:21:47.400758 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:21:47.400775 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:21:47.400806 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:21:47.400816 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:21:47.400923 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 05:21:47.401106 (+ 183us) server_negotiation.cc:299] Negotiation successful
0928 05:21:47.401233 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 05:21:47.402566 18386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:47.393068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33759 (local address 127.0.97.2:48489)
0928 05:21:47.393908 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:47.393941 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 05:21:47.395469 (+ 1528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:21:47.396029 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:21:47.396036 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:21:47.396123 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:21:47.396664 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:47.396680 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:47.397636 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:47.397639 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:47.398813 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:47.398823 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:47.399475 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:47.399482 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:47.399774 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:47.400421 (+ 647us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 05:21:47.400425 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:21:47.400473 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 05:21:47.400580 (+ 107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:21:47.400924 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:21:47.400932 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:21:47.400939 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 05:21:47.401070 (+ 131us) client_negotiation.cc:240] Negotiation successful
0928 05:21:47.401506 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":243,"threads_started":1}
I20260928 05:21:47.403188 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572622576160-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572622576160-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572622576160-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572622576160-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33759
--builtin_ntp_servers=127.0.97.20:38021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 05:21:47.403836 18384 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33759
I20260928 05:21:47.404122 18384 heartbeater.cc:464] Registering TS with master...
I20260928 05:21:47.404906 18384 heartbeater.cc:511] Master 127.0.97.62:33759 requested a full tablet report, sending...
I20260928 05:21:47.406461 18067 ts_manager.cc:194] Registered new tserver with Master: d04f564474294351a9377ed80ae56d69 (127.0.97.2:39289)
I20260928 05:21:47.407512 18067 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:48489
W20260928 05:21:47.715274 18387 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:21:47.715524 18387 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:21:47.715634 18387 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:21:47.722478 18387 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:21:47.722574 18387 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:21:47.722707 18387 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 05:21:47.731194 18387 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572622576160-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572622576160-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572622576160-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33759
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572622576160-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:21:47.732314 18387 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:21:47.734004 18387 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:21:47.745754 18392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:21:47.745791 18393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:21:47.745823 18395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:21:47.746872 18387 server_base.cc:1034] running on GCE node
I20260928 05:21:47.747445 18387 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:21:47.748279 18387 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:21:47.749478 18387 hybrid_clock.cc:648] HybridClock initialized: now 1790572907749428 us; error 43 us; skew 500 ppm
I20260928 05:21:47.749761 18387 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 05:21:47.751739 18387 webserver.cc:460] Webserver started at http://127.0.97.3:35671/ using document root <none> and password file <none>
I20260928 05:21:47.752347 18387 fs_manager.cc:359] Metadata directory not provided
I20260928 05:21:47.752439 18387 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:21:47.752715 18387 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:21:47.755424 18387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572622576160-388-0/minicluster-data/ts-2/data/instance:
uuid: "6df82a41c05b47a7b75efda69ecf4304"
format_stamp: "Formatted at 2026-09-28 05:21:47 on dist-test-slave-whsn"
server_key: "b9edee35d28fedc13440926c068807bb"
server_key_iv: "6eadaf2efd40e1f425c78611024c4276"
server_key_version: "encryptionkey@0"
I20260928 05:21:47.756134 18387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572622576160-388-0/minicluster-data/ts-2/wal/instance:
uuid: "6df82a41c05b47a7b75efda69ecf4304"
format_stamp: "Formatted at 2026-09-28 05:21:47 on dist-test-slave-whsn"
server_key: "b9edee35d28fedc13440926c068807bb"
server_key_iv: "6eadaf2efd40e1f425c78611024c4276"
server_key_version: "encryptionkey@0"
I20260928 05:21:47.760645 18387 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260928 05:21:47.763731 18401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:21:47.764956 18387 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 05:21:47.765115 18387 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572622576160-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572622576160-388-0/minicluster-data/ts-2/wal
uuid: "6df82a41c05b47a7b75efda69ecf4304"
format_stamp: "Formatted at 2026-09-28 05:21:47 on dist-test-slave-whsn"
server_key: "b9edee35d28fedc13440926c068807bb"
server_key_iv: "6eadaf2efd40e1f425c78611024c4276"
server_key_version: "encryptionkey@0"
I20260928 05:21:47.765290 18387 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572622576160-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572622576160-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572622576160-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:21:47.786588 18387 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:21:47.787362 18387 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:21:47.787600 18387 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:21:47.788468 18387 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:21:47.789901 18387 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:21:47.790002 18387 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:21:47.790099 18387 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:21:47.790163 18387 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:21:47.818317 18387 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41157
I20260928 05:21:47.818383 18513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41157 every 8 connection(s)
I20260928 05:21:47.819460 18387 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572622576160-388-0/minicluster-data/ts-2/data/info.pb
I20260928 05:21:47.825088 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 18387
I20260928 05:21:47.825254 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572622576160-388-0/minicluster-data/ts-2/wal/instance
I20260928 05:21:47.825774 388 external_mini_cluster.cc:1428] Setting key 93c7c41ff8a5c7eb1e6ab8462ca22d91
I20260928 05:21:47.831040 18255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:47.821746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44183 (local address 127.0.97.62:33759)
0928 05:21:47.821933 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 05:21:47.821937 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:21:47.823221 (+ 1284us) server_negotiation.cc:407] Connection header received
0928 05:21:47.824977 (+ 1756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:21:47.824982 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:21:47.825048 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:21:47.825134 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:21:47.826398 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:47.827234 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:47.828870 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:47.829294 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:47.829385 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:47.830477 (+ 1092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:21:47.830482 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:21:47.830485 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:21:47.830501 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:21:47.830529 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:21:47.830539 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:21:47.830601 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 05:21:47.830749 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 05:21:47.830815 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 05:21:47.831887 18516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:47.822249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33759 (local address 127.0.97.3:44183)
0928 05:21:47.823076 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:47.823116 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 05:21:47.824668 (+ 1552us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:21:47.825377 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:21:47.825389 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:21:47.825499 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:21:47.826239 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:47.826258 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:47.827375 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:47.827379 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:47.828732 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:47.828743 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:47.829451 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:47.829456 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:47.829644 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:47.830214 (+ 570us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 05:21:47.830216 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:21:47.830251 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 05:21:47.830325 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:21:47.830606 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:21:47.830611 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:21:47.830616 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 05:21:47.830734 (+ 118us) client_negotiation.cc:240] Negotiation successful
0928 05:21:47.830986 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":251,"threads_started":1}
I20260928 05:21:47.833003 18514 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33759
I20260928 05:21:47.833271 18514 heartbeater.cc:464] Registering TS with master...
I20260928 05:21:47.833879 18514 heartbeater.cc:511] Master 127.0.97.62:33759 requested a full tablet report, sending...
I20260928 05:21:47.835083 18067 ts_manager.cc:194] Registered new tserver with Master: 6df82a41c05b47a7b75efda69ecf4304 (127.0.97.3:41157)
I20260928 05:21:47.835863 18067 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:44183
I20260928 05:21:47.840032 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 05:21:47.848747 18255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:47.843689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52098 (local address 127.0.97.62:33759)
0928 05:21:47.843870 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 05:21:47.843877 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:21:47.844199 (+ 322us) server_negotiation.cc:407] Connection header received
0928 05:21:47.844314 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:21:47.844318 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:21:47.844384 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:21:47.844471 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:21:47.844834 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:47.845535 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:47.846788 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:47.847175 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:47.847263 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:47.848078 (+ 815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:21:47.848083 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:21:47.848086 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:21:47.848102 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:21:47.848129 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:21:47.848140 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:21:47.848206 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 05:21:47.848411 (+ 205us) server_negotiation.cc:299] Negotiation successful
0928 05:21:47.848482 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 05:21:47.851977 18067 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52098:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 05:21:47.858709 18067 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 05:21:47.887993 18526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:47.869941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39289 (local address 127.0.0.1:55306)
0928 05:21:47.880524 (+ 10583us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:47.880544 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 05:21:47.880671 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:21:47.881430 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:21:47.881436 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:21:47.881464 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:21:47.881774 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:47.881786 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:47.883652 (+ 1866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:47.883657 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:47.885622 (+ 1965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:47.885634 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:47.887167 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:47.887173 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:47.887363 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:47.887412 (+ 49us) client_negotiation.cc:769] Sending connection context
0928 05:21:47.887530 (+ 118us) client_negotiation.cc:240] Negotiation successful
0928 05:21:47.887703 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10437,"mutex_wait_us":2,"thread_start_us":9599,"threads_started":1}
I20260928 05:21:47.888109 18529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:47.870571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55306 (local address 127.0.97.2:39289)
0928 05:21:47.871160 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0928 05:21:47.871170 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 05:21:47.880806 (+ 9636us) server_negotiation.cc:407] Connection header received
0928 05:21:47.880943 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:21:47.880953 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:21:47.881218 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:21:47.881397 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:21:47.881960 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:47.883526 (+ 1566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:47.885838 (+ 2312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:47.886984 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:47.887122 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:47.887379 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0928 05:21:47.887577 (+ 198us) server_negotiation.cc:299] Negotiation successful
0928 05:21:47.887737 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":256,"threads_started":1}
I20260928 05:21:47.889328 18530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:47.872091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41157 (local address 127.0.0.1:48168)
0928 05:21:47.880794 (+ 8703us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:47.880810 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 05:21:47.882082 (+ 1272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:21:47.882612 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:21:47.882616 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:21:47.882638 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:21:47.882946 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:47.882954 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:47.884501 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:47.884506 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:47.886595 (+ 2089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:47.886608 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:47.888542 (+ 1934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:47.888549 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:47.888706 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:47.888736 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 05:21:47.888824 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 05:21:47.888966 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8600,"mutex_wait_us":41,"thread_start_us":9320,"threads_started":1}
I20260928 05:21:47.889375 18531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:47.872717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48168 (local address 127.0.97.3:41157)
0928 05:21:47.879327 (+ 6610us) server_negotiation.cc:206] Beginning negotiation
0928 05:21:47.879337 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 05:21:47.880916 (+ 1579us) server_negotiation.cc:407] Connection header received
0928 05:21:47.882265 (+ 1349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:21:47.882272 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:21:47.882456 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:21:47.882610 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:21:47.883111 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:47.884363 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:47.886789 (+ 2426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:47.888050 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:47.888181 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:47.888416 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0928 05:21:47.888908 (+ 492us) server_negotiation.cc:299] Negotiation successful
0928 05:21:47.889029 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6443,"thread_start_us":6302,"threads_started":1}
I20260928 05:21:47.889850 18528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:47.869734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43321 (local address 127.0.0.1:57192)
0928 05:21:47.879899 (+ 10165us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:47.879945 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 05:21:47.880098 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:21:47.881842 (+ 1744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:21:47.881846 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:21:47.881869 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:21:47.882204 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:47.882214 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:47.883883 (+ 1669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:47.883888 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:47.885976 (+ 2088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:47.885991 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:47.889198 (+ 3207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:47.889205 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:47.889381 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:47.889413 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 05:21:47.889497 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 05:21:47.889601 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9939,"thread_start_us":10499,"threads_started":1}
I20260928 05:21:47.890596 18527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:47.869913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57192 (local address 127.0.97.1:43321)
0928 05:21:47.881340 (+ 11427us) server_negotiation.cc:206] Beginning negotiation
0928 05:21:47.881348 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:21:47.881374 (+ 26us) server_negotiation.cc:407] Connection header received
0928 05:21:47.881464 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:21:47.881471 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:21:47.881679 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:21:47.881818 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:21:47.882354 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:47.883774 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:47.886184 (+ 2410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:47.887425 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:47.887561 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:47.887767 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0928 05:21:47.890108 (+ 2341us) server_negotiation.cc:299] Negotiation successful
0928 05:21:47.890281 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11276,"thread_start_us":11136,"threads_started":1}
I20260928 05:21:47.891867 18319 tablet_service.cc:1467] Processing CreateTablet for tablet 5ebaac60671d40538f930c75ef2ba912 (DEFAULT_TABLE table=test-table [id=463687f10b774c498cd8cc0aeb69dfdf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:21:47.893340 18319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ebaac60671d40538f930c75ef2ba912. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:21:47.893152 18449 tablet_service.cc:1467] Processing CreateTablet for tablet 5ebaac60671d40538f930c75ef2ba912 (DEFAULT_TABLE table=test-table [id=463687f10b774c498cd8cc0aeb69dfdf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:21:47.894495 18449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ebaac60671d40538f930c75ef2ba912. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:21:47.895419 18188 tablet_service.cc:1467] Processing CreateTablet for tablet 5ebaac60671d40538f930c75ef2ba912 (DEFAULT_TABLE table=test-table [id=463687f10b774c498cd8cc0aeb69dfdf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:21:47.896364 18188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ebaac60671d40538f930c75ef2ba912. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:21:47.907601 18532 tablet_bootstrap.cc:492] T 5ebaac60671d40538f930c75ef2ba912 P 4c4e7bf2fb584a6a898c1efc57215377: Bootstrap starting.
I20260928 05:21:47.909572 18534 tablet_bootstrap.cc:492] T 5ebaac60671d40538f930c75ef2ba912 P 6df82a41c05b47a7b75efda69ecf4304: Bootstrap starting.
I20260928 05:21:47.910965 18532 tablet_bootstrap.cc:654] T 5ebaac60671d40538f930c75ef2ba912 P 4c4e7bf2fb584a6a898c1efc57215377: Neither blocks nor log segments found. Creating new log.
I20260928 05:21:47.912160 18532 log.cc:824] T 5ebaac60671d40538f930c75ef2ba912 P 4c4e7bf2fb584a6a898c1efc57215377: Log is configured to *not* fsync() on all Append() calls
I20260928 05:21:47.912875 18534 tablet_bootstrap.cc:654] T 5ebaac60671d40538f930c75ef2ba912 P 6df82a41c05b47a7b75efda69ecf4304: Neither blocks nor log segments found. Creating new log.
I20260928 05:21:47.913468 18533 tablet_bootstrap.cc:492] T 5ebaac60671d40538f930c75ef2ba912 P d04f564474294351a9377ed80ae56d69: Bootstrap starting.
I20260928 05:21:47.914134 18534 log.cc:824] T 5ebaac60671d40538f930c75ef2ba912 P 6df82a41c05b47a7b75efda69ecf4304: Log is configured to *not* fsync() on all Append() calls
I20260928 05:21:47.915115 18532 tablet_bootstrap.cc:492] T 5ebaac60671d40538f930c75ef2ba912 P 4c4e7bf2fb584a6a898c1efc57215377: No bootstrap required, opened a new log
I20260928 05:21:47.915378 18532 ts_tablet_manager.cc:1397] T 5ebaac60671d40538f930c75ef2ba912 P 4c4e7bf2fb584a6a898c1efc57215377: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260928 05:21:47.916424 18533 tablet_bootstrap.cc:654] T 5ebaac60671d40538f930c75ef2ba912 P d04f564474294351a9377ed80ae56d69: Neither blocks nor log segments found. Creating new log.
I20260928 05:21:47.917084 18534 tablet_bootstrap.cc:492] T 5ebaac60671d40538f930c75ef2ba912 P 6df82a41c05b47a7b75efda69ecf4304: No bootstrap required, opened a new log
I20260928 05:21:47.917373 18534 ts_tablet_manager.cc:1397] T 5ebaac60671d40538f930c75ef2ba912 P 6df82a41c05b47a7b75efda69ecf4304: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260928 05:21:47.917557 18533 log.cc:824] T 5ebaac60671d40538f930c75ef2ba912 P d04f564474294351a9377ed80ae56d69: Log is configured to *not* fsync() on all Append() calls
I20260928 05:21:47.922681 18532 raft_consensus.cc:348] T 5ebaac60671d40538f930c75ef2ba912 P 4c4e7bf2fb584a6a898c1efc57215377 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c4e7bf2fb584a6a898c1efc57215377" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43321 } } peers { permanent_uuid: "d04f564474294351a9377ed80ae56d69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39289 } } peers { permanent_uuid: "6df82a41c05b47a7b75efda69ecf4304" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41157 } }
I20260928 05:21:47.923000 18532 raft_consensus.cc:374] T 5ebaac60671d40538f930c75ef2ba912 P 4c4e7bf2fb584a6a898c1efc57215377 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:21:47.923127 18532 raft_consensus.cc:729] T 5ebaac60671d40538f930c75ef2ba912 P 4c4e7bf2fb584a6a898c1efc57215377 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c4e7bf2fb584a6a898c1efc57215377, State: Initialized, Role: FOLLOWER
I20260928 05:21:47.923326 18533 tablet_bootstrap.cc:492] T 5ebaac60671d40538f930c75ef2ba912 P d04f564474294351a9377ed80ae56d69: No bootstrap required, opened a new log
I20260928 05:21:47.923629 18533 ts_tablet_manager.cc:1397] T 5ebaac60671d40538f930c75ef2ba912 P d04f564474294351a9377ed80ae56d69: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.003s
I20260928 05:21:47.923632 18532 consensus_queue.cc:260] T 5ebaac60671d40538f930c75ef2ba912 P 4c4e7bf2fb584a6a898c1efc57215377 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c4e7bf2fb584a6a898c1efc57215377" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43321 } } peers { permanent_uuid: "d04f564474294351a9377ed80ae56d69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39289 } } peers { permanent_uuid: "6df82a41c05b47a7b75efda69ecf4304" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41157 } }
I20260928 05:21:47.924623 18253 heartbeater.cc:503] Master 127.0.97.62:33759 was elected leader, sending a full tablet report...
I20260928 05:21:47.925119 18532 ts_tablet_manager.cc:1428] T 5ebaac60671d40538f930c75ef2ba912 P 4c4e7bf2fb584a6a898c1efc57215377: Time spent starting tablet: real 0.010s user 0.005s sys 0.006s
I20260928 05:21:47.927946 18534 raft_consensus.cc:348] T 5ebaac60671d40538f930c75ef2ba912 P 6df82a41c05b47a7b75efda69ecf4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c4e7bf2fb584a6a898c1efc57215377" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43321 } } peers { permanent_uuid: "d04f564474294351a9377ed80ae56d69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39289 } } peers { permanent_uuid: "6df82a41c05b47a7b75efda69ecf4304" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41157 } }
I20260928 05:21:47.928339 18534 raft_consensus.cc:374] T 5ebaac60671d40538f930c75ef2ba912 P 6df82a41c05b47a7b75efda69ecf4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:21:47.928426 18534 raft_consensus.cc:729] T 5ebaac60671d40538f930c75ef2ba912 P 6df82a41c05b47a7b75efda69ecf4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6df82a41c05b47a7b75efda69ecf4304, State: Initialized, Role: FOLLOWER
I20260928 05:21:47.928998 18534 consensus_queue.cc:260] T 5ebaac60671d40538f930c75ef2ba912 P 6df82a41c05b47a7b75efda69ecf4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c4e7bf2fb584a6a898c1efc57215377" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43321 } } peers { permanent_uuid: "d04f564474294351a9377ed80ae56d69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39289 } } peers { permanent_uuid: "6df82a41c05b47a7b75efda69ecf4304" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41157 } }
I20260928 05:21:47.929970 18514 heartbeater.cc:503] Master 127.0.97.62:33759 was elected leader, sending a full tablet report...
I20260928 05:21:47.930616 18534 ts_tablet_manager.cc:1428] T 5ebaac60671d40538f930c75ef2ba912 P 6df82a41c05b47a7b75efda69ecf4304: Time spent starting tablet: real 0.013s user 0.011s sys 0.000s
I20260928 05:21:47.933982 18533 raft_consensus.cc:348] T 5ebaac60671d40538f930c75ef2ba912 P d04f564474294351a9377ed80ae56d69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c4e7bf2fb584a6a898c1efc57215377" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43321 } } peers { permanent_uuid: "d04f564474294351a9377ed80ae56d69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39289 } } peers { permanent_uuid: "6df82a41c05b47a7b75efda69ecf4304" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41157 } }
I20260928 05:21:47.934329 18533 raft_consensus.cc:374] T 5ebaac60671d40538f930c75ef2ba912 P d04f564474294351a9377ed80ae56d69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:21:47.934396 18533 raft_consensus.cc:729] T 5ebaac60671d40538f930c75ef2ba912 P d04f564474294351a9377ed80ae56d69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d04f564474294351a9377ed80ae56d69, State: Initialized, Role: FOLLOWER
I20260928 05:21:47.934832 18533 consensus_queue.cc:260] T 5ebaac60671d40538f930c75ef2ba912 P d04f564474294351a9377ed80ae56d69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c4e7bf2fb584a6a898c1efc57215377" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43321 } } peers { permanent_uuid: "d04f564474294351a9377ed80ae56d69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39289 } } peers { permanent_uuid: "6df82a41c05b47a7b75efda69ecf4304" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41157 } }
I20260928 05:21:47.935735 18384 heartbeater.cc:503] Master 127.0.97.62:33759 was elected leader, sending a full tablet report...
I20260928 05:21:47.936280 18533 ts_tablet_manager.cc:1428] T 5ebaac60671d40538f930c75ef2ba912 P d04f564474294351a9377ed80ae56d69: Time spent starting tablet: real 0.013s user 0.011s sys 0.000s
I20260928 05:21:47.964367 18539 raft_consensus.cc:482] T 5ebaac60671d40538f930c75ef2ba912 P 6df82a41c05b47a7b75efda69ecf4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:21:47.964671 18539 raft_consensus.cc:504] T 5ebaac60671d40538f930c75ef2ba912 P 6df82a41c05b47a7b75efda69ecf4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c4e7bf2fb584a6a898c1efc57215377" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43321 } } peers { permanent_uuid: "d04f564474294351a9377ed80ae56d69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39289 } } peers { permanent_uuid: "6df82a41c05b47a7b75efda69ecf4304" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41157 } }
I20260928 05:21:47.966565 18539 leader_election.cc:288] T 5ebaac60671d40538f930c75ef2ba912 P 6df82a41c05b47a7b75efda69ecf4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c4e7bf2fb584a6a898c1efc57215377 (127.0.97.1:43321), d04f564474294351a9377ed80ae56d69 (127.0.97.2:39289)
W20260928 05:21:47.968549 18254 tablet.cc:2369] T 5ebaac60671d40538f930c75ef2ba912 P 4c4e7bf2fb584a6a898c1efc57215377: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 05:21:47.974176 18529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:47.966956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45037 (local address 127.0.97.2:39289)
0928 05:21:47.967176 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0928 05:21:47.967182 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:21:47.967806 (+ 624us) server_negotiation.cc:407] Connection header received
0928 05:21:47.967964 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:21:47.967970 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:21:47.968037 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:21:47.968111 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:21:47.968813 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:47.969842 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:47.972523 (+ 2681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:47.973379 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:47.973489 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:47.973580 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 05:21:47.973825 (+ 245us) server_negotiation.cc:299] Negotiation successful
0928 05:21:47.973944 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260928 05:21:47.974275 18541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:47.967030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39289 (local address 127.0.97.3:45037)
0928 05:21:47.967706 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:47.967723 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 05:21:47.967840 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:21:47.968140 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:21:47.968144 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:21:47.968167 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:21:47.968584 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:47.968597 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:47.969966 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:47.969971 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:47.972324 (+ 2353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:47.972338 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:47.973532 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:47.973538 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:47.973689 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:47.973713 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 05:21:47.973790 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 05:21:47.973909 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"mutex_wait_us":123,"thread_start_us":323,"threads_started":1}
I20260928 05:21:47.974795 18516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:47.966922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43321 (local address 127.0.97.3:44583)
0928 05:21:47.967619 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:47.967639 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 05:21:47.967751 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:21:47.968355 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:21:47.968360 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:21:47.968383 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:21:47.968735 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:47.968746 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:47.970156 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:47.970161 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:47.972337 (+ 2176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:47.972347 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:47.974067 (+ 1720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:47.974074 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:47.974267 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:47.974301 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 05:21:47.974378 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 05:21:47.974511 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"mutex_wait_us":136}
I20260928 05:21:47.975196 18339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ebaac60671d40538f930c75ef2ba912" candidate_uuid: "6df82a41c05b47a7b75efda69ecf4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d04f564474294351a9377ed80ae56d69" is_pre_election: true
I20260928 05:21:47.975382 18527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:47.967320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44583 (local address 127.0.97.1:43321)
0928 05:21:47.967923 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0928 05:21:47.967929 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:21:47.967947 (+ 18us) server_negotiation.cc:407] Connection header received
0928 05:21:47.968005 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:21:47.968010 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:21:47.968099 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:21:47.968202 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:21:47.968927 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:47.970041 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:47.972471 (+ 2430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:47.973766 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:47.973880 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:47.973964 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 05:21:47.975026 (+ 1062us) server_negotiation.cc:299] Negotiation successful
0928 05:21:47.975151 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137}
I20260928 05:21:47.975687 18339 raft_consensus.cc:2454] T 5ebaac60671d40538f930c75ef2ba912 P d04f564474294351a9377ed80ae56d69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6df82a41c05b47a7b75efda69ecf4304 in term 0.
I20260928 05:21:47.976234 18208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ebaac60671d40538f930c75ef2ba912" candidate_uuid: "6df82a41c05b47a7b75efda69ecf4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c4e7bf2fb584a6a898c1efc57215377" is_pre_election: true
I20260928 05:21:47.976387 18404 leader_election.cc:302] T 5ebaac60671d40538f930c75ef2ba912 P 6df82a41c05b47a7b75efda69ecf4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6df82a41c05b47a7b75efda69ecf4304, d04f564474294351a9377ed80ae56d69; no voters:
I20260928 05:21:47.976691 18208 raft_consensus.cc:2454] T 5ebaac60671d40538f930c75ef2ba912 P 4c4e7bf2fb584a6a898c1efc57215377 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6df82a41c05b47a7b75efda69ecf4304 in term 0.
I20260928 05:21:47.976806 18539 raft_consensus.cc:2781] T 5ebaac60671d40538f930c75ef2ba912 P 6df82a41c05b47a7b75efda69ecf4304 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 05:21:47.976974 18539 raft_consensus.cc:482] T 5ebaac60671d40538f930c75ef2ba912 P 6df82a41c05b47a7b75efda69ecf4304 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 05:21:47.977066 18539 raft_consensus.cc:3037] T 5ebaac60671d40538f930c75ef2ba912 P 6df82a41c05b47a7b75efda69ecf4304 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:21:47.979204 18539 raft_consensus.cc:504] T 5ebaac60671d40538f930c75ef2ba912 P 6df82a41c05b47a7b75efda69ecf4304 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c4e7bf2fb584a6a898c1efc57215377" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43321 } } peers { permanent_uuid: "d04f564474294351a9377ed80ae56d69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39289 } } peers { permanent_uuid: "6df82a41c05b47a7b75efda69ecf4304" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41157 } }
I20260928 05:21:47.979737 18539 leader_election.cc:288] T 5ebaac60671d40538f930c75ef2ba912 P 6df82a41c05b47a7b75efda69ecf4304 [CANDIDATE]: Term 1 election: Requested vote from peers 4c4e7bf2fb584a6a898c1efc57215377 (127.0.97.1:43321), d04f564474294351a9377ed80ae56d69 (127.0.97.2:39289)
I20260928 05:21:47.980065 18208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ebaac60671d40538f930c75ef2ba912" candidate_uuid: "6df82a41c05b47a7b75efda69ecf4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c4e7bf2fb584a6a898c1efc57215377"
I20260928 05:21:47.980165 18339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ebaac60671d40538f930c75ef2ba912" candidate_uuid: "6df82a41c05b47a7b75efda69ecf4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d04f564474294351a9377ed80ae56d69"
I20260928 05:21:47.980216 18208 raft_consensus.cc:3037] T 5ebaac60671d40538f930c75ef2ba912 P 4c4e7bf2fb584a6a898c1efc57215377 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:21:47.980358 18339 raft_consensus.cc:3037] T 5ebaac60671d40538f930c75ef2ba912 P d04f564474294351a9377ed80ae56d69 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:21:47.982306 18208 raft_consensus.cc:2454] T 5ebaac60671d40538f930c75ef2ba912 P 4c4e7bf2fb584a6a898c1efc57215377 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6df82a41c05b47a7b75efda69ecf4304 in term 1.
I20260928 05:21:47.982554 18339 raft_consensus.cc:2454] T 5ebaac60671d40538f930c75ef2ba912 P d04f564474294351a9377ed80ae56d69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6df82a41c05b47a7b75efda69ecf4304 in term 1.
I20260928 05:21:47.982620 18402 leader_election.cc:302] T 5ebaac60671d40538f930c75ef2ba912 P 6df82a41c05b47a7b75efda69ecf4304 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c4e7bf2fb584a6a898c1efc57215377, 6df82a41c05b47a7b75efda69ecf4304; no voters:
I20260928 05:21:47.982908 18539 raft_consensus.cc:2781] T 5ebaac60671d40538f930c75ef2ba912 P 6df82a41c05b47a7b75efda69ecf4304 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:21:47.983299 18539 raft_consensus.cc:686] T 5ebaac60671d40538f930c75ef2ba912 P 6df82a41c05b47a7b75efda69ecf4304 [term 1 LEADER]: Becoming Leader. State: Replica: 6df82a41c05b47a7b75efda69ecf4304, State: Running, Role: LEADER
I20260928 05:21:47.983780 18539 consensus_queue.cc:237] T 5ebaac60671d40538f930c75ef2ba912 P 6df82a41c05b47a7b75efda69ecf4304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c4e7bf2fb584a6a898c1efc57215377" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43321 } } peers { permanent_uuid: "d04f564474294351a9377ed80ae56d69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39289 } } peers { permanent_uuid: "6df82a41c05b47a7b75efda69ecf4304" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41157 } }
I20260928 05:21:47.988276 18067 catalog_manager.cc:5495] T 5ebaac60671d40538f930c75ef2ba912 P 6df82a41c05b47a7b75efda69ecf4304 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6df82a41c05b47a7b75efda69ecf4304 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "6df82a41c05b47a7b75efda69ecf4304" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c4e7bf2fb584a6a898c1efc57215377" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d04f564474294351a9377ed80ae56d69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6df82a41c05b47a7b75efda69ecf4304" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41157 } health_report { overall_health: HEALTHY } } }
I20260928 05:21:48.030747 18531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:48.025728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48174 (local address 127.0.97.3:41157)
0928 05:21:48.025872 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 05:21:48.025877 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:21:48.025892 (+ 15us) server_negotiation.cc:407] Connection header received
0928 05:21:48.025971 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:21:48.025974 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:21:48.026062 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:21:48.026180 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:21:48.026510 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:48.027217 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:48.028662 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:48.029160 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:48.029259 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:48.029722 (+ 463us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:21:48.030241 (+ 519us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:21:48.030345 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 05:21:48.030443 (+ 98us) server_negotiation.cc:299] Negotiation successful
0928 05:21:48.030510 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 05:21:48.038168 18339 raft_consensus.cc:1261] T 5ebaac60671d40538f930c75ef2ba912 P d04f564474294351a9377ed80ae56d69 [term 1 FOLLOWER]: Refusing update from remote peer 6df82a41c05b47a7b75efda69ecf4304: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:21:48.038196 18208 raft_consensus.cc:1261] T 5ebaac60671d40538f930c75ef2ba912 P 4c4e7bf2fb584a6a898c1efc57215377 [term 1 FOLLOWER]: Refusing update from remote peer 6df82a41c05b47a7b75efda69ecf4304: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:21:48.038977 18542 consensus_queue.cc:1035] T 5ebaac60671d40538f930c75ef2ba912 P 6df82a41c05b47a7b75efda69ecf4304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d04f564474294351a9377ed80ae56d69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:21:48.039363 18539 consensus_queue.cc:1035] T 5ebaac60671d40538f930c75ef2ba912 P 6df82a41c05b47a7b75efda69ecf4304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c4e7bf2fb584a6a898c1efc57215377" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:21:48.060256 18066 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52098:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:21:48.060657 18547 mvcc.cc:205] Tried to move back new op lower bound from 7334186631315324928 to 7334186631105359872. Current Snapshot: MvccSnapshot[applied={T|T < 7334186631315324928}]
I20260928 05:21:48.060721 18066 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52098:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:21:48.064373 18066 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7add23e8d8b649bdbeac9df8e437a966: Sending DeleteTablet for 3 replicas of tablet 5ebaac60671d40538f930c75ef2ba912
I20260928 05:21:48.065399 18188 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ebaac60671d40538f930c75ef2ba912 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:21:48 UTC) from {username='slave'} at 127.0.0.1:57192
I20260928 05:21:48.065589 18319 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ebaac60671d40538f930c75ef2ba912 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:21:48 UTC) from {username='slave'} at 127.0.0.1:55306
I20260928 05:21:48.066146 18554 tablet_replica.cc:330] T 5ebaac60671d40538f930c75ef2ba912 P 4c4e7bf2fb584a6a898c1efc57215377: stopping tablet replica
I20260928 05:21:48.066433 18449 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ebaac60671d40538f930c75ef2ba912 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:21:48 UTC) from {username='slave'} at 127.0.0.1:48168
I20260928 05:21:48.066551 18554 raft_consensus.cc:2229] T 5ebaac60671d40538f930c75ef2ba912 P 4c4e7bf2fb584a6a898c1efc57215377 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:21:48.066964 18554 raft_consensus.cc:2258] T 5ebaac60671d40538f930c75ef2ba912 P 4c4e7bf2fb584a6a898c1efc57215377 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:21:48.067492 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 18126
I20260928 05:21:48.076887 18554 ts_tablet_manager.cc:1905] T 5ebaac60671d40538f930c75ef2ba912 P 4c4e7bf2fb584a6a898c1efc57215377: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:21:48.077108 18556 tablet_replica.cc:330] T 5ebaac60671d40538f930c75ef2ba912 P 6df82a41c05b47a7b75efda69ecf4304: stopping tablet replica
I20260928 05:21:48.077591 18556 raft_consensus.cc:2229] T 5ebaac60671d40538f930c75ef2ba912 P 6df82a41c05b47a7b75efda69ecf4304 [term 1 LEADER]: Raft consensus shutting down.
I20260928 05:21:48.078704 18556 raft_consensus.cc:2258] T 5ebaac60671d40538f930c75ef2ba912 P 6df82a41c05b47a7b75efda69ecf4304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:21:48.081756 18556 ts_tablet_manager.cc:1905] T 5ebaac60671d40538f930c75ef2ba912 P 6df82a41c05b47a7b75efda69ecf4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:21:48.083365 18554 ts_tablet_manager.cc:1918] T 5ebaac60671d40538f930c75ef2ba912 P 4c4e7bf2fb584a6a898c1efc57215377: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:21:48.083738 18554 log.cc:1196] T 5ebaac60671d40538f930c75ef2ba912 P 4c4e7bf2fb584a6a898c1efc57215377: Deleting WAL directory at /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572622576160-388-0/minicluster-data/ts-0/wal/wals/5ebaac60671d40538f930c75ef2ba912
I20260928 05:21:48.084251 18554 ts_tablet_manager.cc:1939] T 5ebaac60671d40538f930c75ef2ba912 P 4c4e7bf2fb584a6a898c1efc57215377: Deleting consensus metadata
I20260928 05:21:48.084802 18555 tablet_replica.cc:330] T 5ebaac60671d40538f930c75ef2ba912 P d04f564474294351a9377ed80ae56d69: stopping tablet replica
I20260928 05:21:48.085378 18555 raft_consensus.cc:2229] T 5ebaac60671d40538f930c75ef2ba912 P d04f564474294351a9377ed80ae56d69 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:21:48.085893 18555 raft_consensus.cc:2258] T 5ebaac60671d40538f930c75ef2ba912 P d04f564474294351a9377ed80ae56d69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:21:48.086022 18051 catalog_manager.cc:4841] TS 4c4e7bf2fb584a6a898c1efc57215377 (127.0.97.1:43321): tablet 5ebaac60671d40538f930c75ef2ba912 (table test-table [id=463687f10b774c498cd8cc0aeb69dfdf]) successfully deleted
I20260928 05:21:48.087930 18555 ts_tablet_manager.cc:1905] T 5ebaac60671d40538f930c75ef2ba912 P d04f564474294351a9377ed80ae56d69: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:21:48.090291 18556 ts_tablet_manager.cc:1918] T 5ebaac60671d40538f930c75ef2ba912 P 6df82a41c05b47a7b75efda69ecf4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:21:48.090435 18556 log.cc:1196] T 5ebaac60671d40538f930c75ef2ba912 P 6df82a41c05b47a7b75efda69ecf4304: Deleting WAL directory at /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572622576160-388-0/minicluster-data/ts-2/wal/wals/5ebaac60671d40538f930c75ef2ba912
I20260928 05:21:48.090893 18556 ts_tablet_manager.cc:1939] T 5ebaac60671d40538f930c75ef2ba912 P 6df82a41c05b47a7b75efda69ecf4304: Deleting consensus metadata
I20260928 05:21:48.092307 18054 catalog_manager.cc:4841] TS 6df82a41c05b47a7b75efda69ecf4304 (127.0.97.3:41157): tablet 5ebaac60671d40538f930c75ef2ba912 (table test-table [id=463687f10b774c498cd8cc0aeb69dfdf]) successfully deleted
I20260928 05:21:48.094501 18555 ts_tablet_manager.cc:1918] T 5ebaac60671d40538f930c75ef2ba912 P d04f564474294351a9377ed80ae56d69: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:21:48.094653 18555 log.cc:1196] T 5ebaac60671d40538f930c75ef2ba912 P d04f564474294351a9377ed80ae56d69: Deleting WAL directory at /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572622576160-388-0/minicluster-data/ts-1/wal/wals/5ebaac60671d40538f930c75ef2ba912
I20260928 05:21:48.095198 18555 ts_tablet_manager.cc:1939] T 5ebaac60671d40538f930c75ef2ba912 P d04f564474294351a9377ed80ae56d69: Deleting consensus metadata
I20260928 05:21:48.095404 18527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:48.068499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57200 (local address 127.0.97.1:43321)
0928 05:21:48.068650 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 05:21:48.068655 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:21:48.078137 (+ 9482us) server_negotiation.cc:407] Connection header received
0928 05:21:48.079109 (+ 972us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:21:48.079116 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:21:48.079230 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:21:48.079302 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:21:48.081944 (+ 2642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:48.083053 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:48.092187 (+ 9134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:48.092726 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:48.092801 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:48.094029 (+ 1228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:21:48.094038 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:21:48.094049 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:21:48.094577 (+ 528us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:21:48.094632 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:21:48.094655 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:21:48.094761 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 05:21:48.094954 (+ 193us) server_negotiation.cc:299] Negotiation successful
0928 05:21:48.095068 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 05:21:48.096108 18248 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:21:48.097061 18053 catalog_manager.cc:4841] TS d04f564474294351a9377ed80ae56d69 (127.0.97.2:39289): tablet 5ebaac60671d40538f930c75ef2ba912 (table test-table [id=463687f10b774c498cd8cc0aeb69dfdf]) successfully deleted
I20260928 05:21:48.184360 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 18126
I20260928 05:21:48.204874 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 18257
I20260928 05:21:48.211503 18529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:48.205632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55310 (local address 127.0.97.2:39289)
0928 05:21:48.205829 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 05:21:48.205834 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:21:48.205853 (+ 19us) server_negotiation.cc:407] Connection header received
0928 05:21:48.206016 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:21:48.206023 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:21:48.206125 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:21:48.206210 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:21:48.206586 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:48.207349 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:48.208596 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:48.209141 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:48.209247 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:48.210246 (+ 999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:21:48.210254 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:21:48.210268 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:21:48.210819 (+ 551us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:21:48.210862 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:21:48.210881 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:21:48.210973 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 05:21:48.211100 (+ 127us) server_negotiation.cc:299] Negotiation successful
0928 05:21:48.211203 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 05:21:48.212117 18379 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:21:48.312150 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 18257
I20260928 05:21:48.325811 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 18387
I20260928 05:21:48.332504 18531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:48.326505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48182 (local address 127.0.97.3:41157)
0928 05:21:48.326671 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 05:21:48.326692 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 05:21:48.326743 (+ 51us) server_negotiation.cc:407] Connection header received
0928 05:21:48.326956 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:21:48.326960 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:21:48.327043 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:21:48.327130 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:21:48.327517 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:48.328332 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:48.329663 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:48.330099 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:48.330207 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:48.331195 (+ 988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:21:48.331203 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:21:48.331219 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:21:48.331802 (+ 583us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:21:48.331847 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:21:48.331870 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:21:48.331967 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 05:21:48.332098 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 05:21:48.332198 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 05:21:48.333201 18509 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:21:48.457751 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 18387
I20260928 05:21:48.473363 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 18036
I20260928 05:21:48.474334 18097 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:21:48.580534 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 18036
2026-09-28T05:21:48Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2600 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572622576160-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:21:48 dist-test-slave-whsn krb5kdc[18564](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:21:48 dist-test-slave-whsn krb5kdc[18564](info): set up 2 sockets
Sep 28 05:21:48 dist-test-slave-whsn krb5kdc[18564](info): commencing operation
krb5kdc: starting...
W20260928 05:21:49.588826 388 mini_kdc.cc:121] Time spent starting KDC: real 0.972s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572622576160-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572622576160-388-0/krb5kdc/test-user.keytab.
Sep 28 05:21:49 dist-test-slave-whsn krb5kdc[18564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572909, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:21:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:21:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 05:21:49.798194 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572622576160-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572622576160-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572622576160-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44433
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44433
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:21:50.103037 18577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:21:50.103292 18577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:21:50.103416 18577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:21:50.110355 18577 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:21:50.110445 18577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:21:50.110502 18577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:21:50.110584 18577 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:21:50.110638 18577 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:21:50.119926 18577 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572622576160-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572622576160-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44433
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44433
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572622576160-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:21:50.121138 18577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:21:50.122383 18577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:21:50.132915 18582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:21:50.132949 18583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:21:50.132995 18585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:21:50.133693 18577 server_base.cc:1034] running on GCE node
I20260928 05:21:50.134315 18577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:21:50.135495 18577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:21:50.136693 18577 hybrid_clock.cc:648] HybridClock initialized: now 1790572910136657 us; error 26 us; skew 500 ppm
I20260928 05:21:50.137022 18577 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:21:50 dist-test-slave-whsn krb5kdc[18564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572910, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:21:50.139806 18577 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 05:21:50.141170 18577 webserver.cc:460] Webserver started at http://127.0.97.62:38393/ using document root <none> and password file <none>
I20260928 05:21:50.141768 18577 fs_manager.cc:359] Metadata directory not provided
I20260928 05:21:50.141866 18577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:21:50.142136 18577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:21:50.144833 18577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572622576160-388-0/minicluster-data/master-0/data/instance:
uuid: "549e378ef8a743fea5eeec8560f320ff"
format_stamp: "Formatted at 2026-09-28 05:21:50 on dist-test-slave-whsn"
server_key: "7f75f138467ce8a207be75f83b078d13"
server_key_iv: "a6935935f7d6a39023421b6ffe3a9613"
server_key_version: "encryptionkey@0"
I20260928 05:21:50.145576 18577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572622576160-388-0/minicluster-data/master-0/wal/instance:
uuid: "549e378ef8a743fea5eeec8560f320ff"
format_stamp: "Formatted at 2026-09-28 05:21:50 on dist-test-slave-whsn"
server_key: "7f75f138467ce8a207be75f83b078d13"
server_key_iv: "a6935935f7d6a39023421b6ffe3a9613"
server_key_version: "encryptionkey@0"
I20260928 05:21:50.150130 18577 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260928 05:21:50.153188 18592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:21:50.154346 18577 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260928 05:21:50.154488 18577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572622576160-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572622576160-388-0/minicluster-data/master-0/wal
uuid: "549e378ef8a743fea5eeec8560f320ff"
format_stamp: "Formatted at 2026-09-28 05:21:50 on dist-test-slave-whsn"
server_key: "7f75f138467ce8a207be75f83b078d13"
server_key_iv: "a6935935f7d6a39023421b6ffe3a9613"
server_key_version: "encryptionkey@0"
I20260928 05:21:50.154656 18577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572622576160-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572622576160-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572622576160-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:21:50.174418 18577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:21:50.178049 18577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:21:50.178273 18577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:21:50.194387 18577 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44433
I20260928 05:21:50.194422 18643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44433 every 8 connection(s)
I20260928 05:21:50.195679 18577 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:21:50.199127 18644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:21:50.205718 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 18577
I20260928 05:21:50.205871 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572622576160-388-0/minicluster-data/master-0/wal/instance
I20260928 05:21:50.206261 388 external_mini_cluster.cc:1428] Setting key 555fdb126c56c2882d945fd2112da739
I20260928 05:21:50.208760 18644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 549e378ef8a743fea5eeec8560f320ff: Bootstrap starting.
I20260928 05:21:50.211908 18644 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 549e378ef8a743fea5eeec8560f320ff: Neither blocks nor log segments found. Creating new log.
I20260928 05:21:50.213092 18644 log.cc:824] T 00000000000000000000000000000000 P 549e378ef8a743fea5eeec8560f320ff: Log is configured to *not* fsync() on all Append() calls
I20260928 05:21:50.215987 18644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 549e378ef8a743fea5eeec8560f320ff: No bootstrap required, opened a new log
Sep 28 05:21:50 dist-test-slave-whsn krb5kdc[18564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572909, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:21:50.224402 18644 raft_consensus.cc:348] T 00000000000000000000000000000000 P 549e378ef8a743fea5eeec8560f320ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "549e378ef8a743fea5eeec8560f320ff" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44433 } }
I20260928 05:21:50.224812 18644 raft_consensus.cc:374] T 00000000000000000000000000000000 P 549e378ef8a743fea5eeec8560f320ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:21:50.224954 18644 raft_consensus.cc:729] T 00000000000000000000000000000000 P 549e378ef8a743fea5eeec8560f320ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 549e378ef8a743fea5eeec8560f320ff, State: Initialized, Role: FOLLOWER
I20260928 05:21:50.225546 18644 consensus_queue.cc:260] T 00000000000000000000000000000000 P 549e378ef8a743fea5eeec8560f320ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "549e378ef8a743fea5eeec8560f320ff" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44433 } }
I20260928 05:21:50.225791 18644 raft_consensus.cc:388] T 00000000000000000000000000000000 P 549e378ef8a743fea5eeec8560f320ff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:21:50.225924 18644 raft_consensus.cc:482] T 00000000000000000000000000000000 P 549e378ef8a743fea5eeec8560f320ff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:21:50.226089 18644 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 549e378ef8a743fea5eeec8560f320ff [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:21:50.228107 18647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:50.207886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46182 (local address 127.0.97.62:44433)
0928 05:21:50.208603 (+ 717us) server_negotiation.cc:206] Beginning negotiation
0928 05:21:50.208614 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 05:21:50.208652 (+ 38us) server_negotiation.cc:407] Connection header received
0928 05:21:50.210319 (+ 1667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:21:50.210338 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:21:50.210672 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:21:50.210951 (+ 279us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:21:50.212611 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:50.214222 (+ 1611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:50.216020 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:50.216669 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:50.216823 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:50.221363 (+ 4540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:21:50.221370 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:21:50.221383 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:21:50.221418 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:21:50.224199 (+ 2781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:21:50.224980 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:21:50.224989 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:21:50.224997 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:21:50.225181 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:21:50.225592 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:21:50.225595 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:21:50.225597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:21:50.226162 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:21:50.226386 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0928 05:21:50.226792 (+ 406us) server_negotiation.cc:299] Negotiation successful
0928 05:21:50.227138 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":325,"threads_started":1}
I20260928 05:21:50.228592 18644 raft_consensus.cc:504] T 00000000000000000000000000000000 P 549e378ef8a743fea5eeec8560f320ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "549e378ef8a743fea5eeec8560f320ff" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44433 } }
I20260928 05:21:50.229089 18644 leader_election.cc:302] T 00000000000000000000000000000000 P 549e378ef8a743fea5eeec8560f320ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 549e378ef8a743fea5eeec8560f320ff; no voters:
I20260928 05:21:50.229671 18644 leader_election.cc:288] T 00000000000000000000000000000000 P 549e378ef8a743fea5eeec8560f320ff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:21:50.229753 18649 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 549e378ef8a743fea5eeec8560f320ff [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:21:50.230957 18649 raft_consensus.cc:686] T 00000000000000000000000000000000 P 549e378ef8a743fea5eeec8560f320ff [term 1 LEADER]: Becoming Leader. State: Replica: 549e378ef8a743fea5eeec8560f320ff, State: Running, Role: LEADER
I20260928 05:21:50.231600 18644 sys_catalog.cc:564] T 00000000000000000000000000000000 P 549e378ef8a743fea5eeec8560f320ff [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:21:50.231520 18649 consensus_queue.cc:237] T 00000000000000000000000000000000 P 549e378ef8a743fea5eeec8560f320ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "549e378ef8a743fea5eeec8560f320ff" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44433 } }
I20260928 05:21:50.234265 18650 sys_catalog.cc:455] T 00000000000000000000000000000000 P 549e378ef8a743fea5eeec8560f320ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "549e378ef8a743fea5eeec8560f320ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "549e378ef8a743fea5eeec8560f320ff" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44433 } } }
I20260928 05:21:50.234596 18650 sys_catalog.cc:458] T 00000000000000000000000000000000 P 549e378ef8a743fea5eeec8560f320ff [sys.catalog]: This master's current role is: LEADER
I20260928 05:21:50.234995 18651 sys_catalog.cc:455] T 00000000000000000000000000000000 P 549e378ef8a743fea5eeec8560f320ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 549e378ef8a743fea5eeec8560f320ff. Latest consensus state: current_term: 1 leader_uuid: "549e378ef8a743fea5eeec8560f320ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "549e378ef8a743fea5eeec8560f320ff" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44433 } } }
I20260928 05:21:50.235296 18651 sys_catalog.cc:458] T 00000000000000000000000000000000 P 549e378ef8a743fea5eeec8560f320ff [sys.catalog]: This master's current role is: LEADER
I20260928 05:21:50.235481 18658 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:21:50.240728 18658 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:21:50.249292 18658 catalog_manager.cc:1347] Generated new cluster ID: 96877171ef5a4a2cbc44d4c603f4d4c8
I20260928 05:21:50.249378 18658 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:21:50.262574 18658 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:21:50.264000 18658 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:21:50.280217 18658 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 549e378ef8a743fea5eeec8560f320ff: Generated new TSK 0
I20260928 05:21:50.281026 18658 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572622576160-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572622576160-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260928 05:21:50.332360 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572622576160-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572622576160-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572622576160-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572622576160-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44433
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572622576160-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:21:50.646672 18670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:21:50.646961 18670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:21:50.647076 18670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:21:50.654165 18670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:21:50.654260 18670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:21:50.654366 18670 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 05:21:50.663702 18670 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572622576160-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572622576160-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572622576160-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572622576160-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44433
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572622576160-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:21:50.664924 18670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:21:50.666199 18670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:21:50.679168 18675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:21:50.679134 18678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:21:50.679196 18676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:21:50.680266 18670 server_base.cc:1034] running on GCE node
I20260928 05:21:50.680796 18670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:21:50.681700 18670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:21:50.682883 18670 hybrid_clock.cc:648] HybridClock initialized: now 1790572910682838 us; error 44 us; skew 500 ppm
I20260928 05:21:50.683169 18670 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:21:50 dist-test-slave-whsn krb5kdc[18564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572910, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:21:50.686057 18670 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260928 05:21:50.687373 18670 webserver.cc:460] Webserver started at http://127.0.97.1:37477/ using document root <none> and password file <none>
I20260928 05:21:50.687963 18670 fs_manager.cc:359] Metadata directory not provided
I20260928 05:21:50.688062 18670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:21:50.688301 18670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:21:50.691053 18670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572622576160-388-0/minicluster-data/ts-0/data/instance:
uuid: "9610853da7d24baea0389d4316e3086b"
format_stamp: "Formatted at 2026-09-28 05:21:50 on dist-test-slave-whsn"
server_key: "fc8f97639244edd74e5cb26cabc7bf60"
server_key_iv: "7ab2db0614b5b113d40267e62ca2219e"
server_key_version: "encryptionkey@0"
I20260928 05:21:50.691727 18670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572622576160-388-0/minicluster-data/ts-0/wal/instance:
uuid: "9610853da7d24baea0389d4316e3086b"
format_stamp: "Formatted at 2026-09-28 05:21:50 on dist-test-slave-whsn"
server_key: "fc8f97639244edd74e5cb26cabc7bf60"
server_key_iv: "7ab2db0614b5b113d40267e62ca2219e"
server_key_version: "encryptionkey@0"
I20260928 05:21:50.696393 18670 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 05:21:50.699648 18685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:21:50.701018 18670 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260928 05:21:50.701161 18670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572622576160-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572622576160-388-0/minicluster-data/ts-0/wal
uuid: "9610853da7d24baea0389d4316e3086b"
format_stamp: "Formatted at 2026-09-28 05:21:50 on dist-test-slave-whsn"
server_key: "fc8f97639244edd74e5cb26cabc7bf60"
server_key_iv: "7ab2db0614b5b113d40267e62ca2219e"
server_key_version: "encryptionkey@0"
I20260928 05:21:50.701326 18670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572622576160-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572622576160-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572622576160-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:21:50.728646 18670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:21:50.732476 18670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:21:50.732722 18670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:21:50.733675 18670 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:21:50.734889 18670 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:21:50.734962 18670 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:21:50.735081 18670 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:21:50.735140 18670 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:21:50.764142 18670 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35993
I20260928 05:21:50.764209 18797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35993 every 8 connection(s)
I20260928 05:21:50.765360 18670 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572622576160-388-0/minicluster-data/ts-0/data/info.pb
I20260928 05:21:50.769151 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 18670
I20260928 05:21:50.769347 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572622576160-388-0/minicluster-data/ts-0/wal/instance
I20260928 05:21:50.769800 388 external_mini_cluster.cc:1428] Setting key d6a5bd49b86ec7fd6476984681ed954a
Sep 28 05:21:50 dist-test-slave-whsn krb5kdc[18564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572910, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:21:50.789558 18800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:50.767762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52137 (local address 127.0.97.62:44433)
0928 05:21:50.768169 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0928 05:21:50.768173 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:21:50.769037 (+ 864us) server_negotiation.cc:407] Connection header received
0928 05:21:50.770896 (+ 1859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:21:50.770902 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:21:50.770977 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:21:50.771074 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:21:50.772438 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:50.773257 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:50.774709 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:50.775096 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:50.775202 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:50.783775 (+ 8573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:21:50.783780 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:21:50.783784 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:21:50.783815 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:21:50.786392 (+ 2577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:21:50.787259 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:21:50.787263 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:21:50.787265 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:21:50.787386 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:21:50.787844 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:21:50.787848 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:21:50.787851 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:21:50.788127 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:21:50.788253 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 05:21:50.789110 (+ 857us) server_negotiation.cc:299] Negotiation successful
0928 05:21:50.789296 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":214,"threads_started":1}
I20260928 05:21:50.791204 18801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:50.768220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44433 (local address 127.0.97.1:52137)
0928 05:21:50.768901 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:50.768939 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 05:21:50.770645 (+ 1706us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:21:50.771299 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:21:50.771307 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:21:50.771761 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:21:50.772279 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:50.772296 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:50.773393 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:50.773396 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:50.774581 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:50.774592 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:50.775299 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:50.775307 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:50.775535 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:50.781005 (+ 5470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:21:50.781054 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:21:50.783531 (+ 2477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:21:50.786555 (+ 3024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:21:50.786564 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:21:50.786581 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:21:50.787152 (+ 571us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:21:50.787477 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:21:50.787482 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:21:50.787485 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:21:50.787731 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:21:50.788253 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:21:50.788261 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:21:50.788713 (+ 452us) client_negotiation.cc:769] Sending connection context
0928 05:21:50.789063 (+ 350us) client_negotiation.cc:240] Negotiation successful
0928 05:21:50.789633 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":269,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260928 05:21:50.792769 18798 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44433
I20260928 05:21:50.793081 18798 heartbeater.cc:464] Registering TS with master...
I20260928 05:21:50.793757 18798 heartbeater.cc:511] Master 127.0.97.62:44433 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260928 05:21:50.795907 18609 ts_manager.cc:194] Registered new tserver with Master: 9610853da7d24baea0389d4316e3086b (127.0.97.1:35993)
I20260928 05:21:50.797806 18609 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:52137
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572622576160-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572622576160-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260928 05:21:50.819705 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572622576160-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572622576160-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572622576160-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572622576160-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44433
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572622576160-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:21:51.136371 18804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:21:51.136637 18804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:21:51.136787 18804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:21:51.143877 18804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:21:51.144011 18804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:21:51.144166 18804 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 05:21:51.153568 18804 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572622576160-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572622576160-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572622576160-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572622576160-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44433
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572622576160-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:21:51.154722 18804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:21:51.156050 18804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:21:51.168900 18812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:21:51.168946 18810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:21:51.168977 18809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:21:51.169658 18804 server_base.cc:1034] running on GCE node
I20260928 05:21:51.170166 18804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:21:51.170996 18804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:21:51.172183 18804 hybrid_clock.cc:648] HybridClock initialized: now 1790572911172136 us; error 41 us; skew 500 ppm
I20260928 05:21:51.172461 18804 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:21:51 dist-test-slave-whsn krb5kdc[18564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572911, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:21:51.175390 18804 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260928 05:21:51.176795 18804 webserver.cc:460] Webserver started at http://127.0.97.2:34307/ using document root <none> and password file <none>
I20260928 05:21:51.177445 18804 fs_manager.cc:359] Metadata directory not provided
I20260928 05:21:51.177564 18804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:21:51.177834 18804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:21:51.180481 18804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572622576160-388-0/minicluster-data/ts-1/data/instance:
uuid: "401cfcf03bd54a4f90027259e5c5ec38"
format_stamp: "Formatted at 2026-09-28 05:21:51 on dist-test-slave-whsn"
server_key: "debd0dae5c18c1031c50aa1d42d3767f"
server_key_iv: "1de7c0eef0ccac596b5b81f9e9e54600"
server_key_version: "encryptionkey@0"
I20260928 05:21:51.181231 18804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572622576160-388-0/minicluster-data/ts-1/wal/instance:
uuid: "401cfcf03bd54a4f90027259e5c5ec38"
format_stamp: "Formatted at 2026-09-28 05:21:51 on dist-test-slave-whsn"
server_key: "debd0dae5c18c1031c50aa1d42d3767f"
server_key_iv: "1de7c0eef0ccac596b5b81f9e9e54600"
server_key_version: "encryptionkey@0"
I20260928 05:21:51.185881 18804 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 05:21:51.189062 18819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:21:51.190222 18804 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260928 05:21:51.190361 18804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572622576160-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572622576160-388-0/minicluster-data/ts-1/wal
uuid: "401cfcf03bd54a4f90027259e5c5ec38"
format_stamp: "Formatted at 2026-09-28 05:21:51 on dist-test-slave-whsn"
server_key: "debd0dae5c18c1031c50aa1d42d3767f"
server_key_iv: "1de7c0eef0ccac596b5b81f9e9e54600"
server_key_version: "encryptionkey@0"
I20260928 05:21:51.190551 18804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572622576160-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572622576160-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572622576160-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:21:51.223171 18804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:21:51.226871 18804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:21:51.227111 18804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:21:51.227972 18804 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:21:51.229244 18804 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:21:51.229321 18804 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:21:51.229441 18804 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:21:51.229493 18804 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:21:51.258767 18804 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43083
I20260928 05:21:51.258864 18931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43083 every 8 connection(s)
I20260928 05:21:51.259972 18804 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572622576160-388-0/minicluster-data/ts-1/data/info.pb
I20260928 05:21:51.269626 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 18804
I20260928 05:21:51.269765 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572622576160-388-0/minicluster-data/ts-1/wal/instance
I20260928 05:21:51.270270 388 external_mini_cluster.cc:1428] Setting key f49727847632eb29367a803768f95c55
Sep 28 05:21:51 dist-test-slave-whsn krb5kdc[18564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572911, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:21:51.279042 18800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:51.262421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56343 (local address 127.0.97.62:44433)
0928 05:21:51.262633 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0928 05:21:51.262640 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:21:51.263781 (+ 1141us) server_negotiation.cc:407] Connection header received
0928 05:21:51.265544 (+ 1763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:21:51.265549 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:21:51.265621 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:21:51.265714 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:21:51.267062 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:51.267825 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:51.269361 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:51.269823 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:51.269937 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:51.273622 (+ 3685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:21:51.273627 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:21:51.273630 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:21:51.273663 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:21:51.276281 (+ 2618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:21:51.276955 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:21:51.276960 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:21:51.276964 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:21:51.277090 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:21:51.277495 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:21:51.277499 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:21:51.277501 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:21:51.277736 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:21:51.277864 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 05:21:51.278505 (+ 641us) server_negotiation.cc:299] Negotiation successful
0928 05:21:51.278728 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 05:21:51.279891 18934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:51.262927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44433 (local address 127.0.97.2:56343)
0928 05:21:51.263617 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:51.263664 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 05:21:51.265296 (+ 1632us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:21:51.265898 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:21:51.265906 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:21:51.266364 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:21:51.266916 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:51.266933 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:51.267962 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:51.267967 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:51.269230 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:51.269242 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:51.270036 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:51.270043 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:51.270319 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:51.271019 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:21:51.271049 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:21:51.273314 (+ 2265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:21:51.276416 (+ 3102us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:21:51.276425 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:21:51.276440 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:21:51.276818 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:21:51.277200 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:21:51.277205 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:21:51.277209 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:21:51.277377 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:21:51.277851 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:21:51.277858 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:21:51.278160 (+ 302us) client_negotiation.cc:769] Sending connection context
0928 05:21:51.278385 (+ 225us) client_negotiation.cc:240] Negotiation successful
0928 05:21:51.278857 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":265,"threads_started":1}
I20260928 05:21:51.281235 18932 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44433
I20260928 05:21:51.281523 18932 heartbeater.cc:464] Registering TS with master...
I20260928 05:21:51.282281 18932 heartbeater.cc:511] Master 127.0.97.62:44433 requested a full tablet report, sending...
I20260928 05:21:51.283633 18609 ts_manager.cc:194] Registered new tserver with Master: 401cfcf03bd54a4f90027259e5c5ec38 (127.0.97.2:43083)
I20260928 05:21:51.284380 18609 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:56343
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572622576160-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572622576160-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260928 05:21:51.315639 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572622576160-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572622576160-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572622576160-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572622576160-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44433
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572622576160-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:21:51.617810 18937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:21:51.618067 18937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:21:51.618162 18937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:21:51.625118 18937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:21:51.625214 18937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:21:51.625329 18937 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 05:21:51.634719 18937 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572622576160-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572622576160-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572622576160-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572622576160-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44433
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572622576160-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:21:51.635845 18937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:21:51.637146 18937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:21:51.649155 18943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:21:51.649293 18945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:21:51.649224 18942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:21:51.650079 18937 server_base.cc:1034] running on GCE node
I20260928 05:21:51.650642 18937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:21:51.651491 18937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:21:51.652684 18937 hybrid_clock.cc:648] HybridClock initialized: now 1790572911652642 us; error 31 us; skew 500 ppm
I20260928 05:21:51.653017 18937 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:21:51 dist-test-slave-whsn krb5kdc[18564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572911, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:21:51.659888 18937 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260928 05:21:51.661441 18937 webserver.cc:460] Webserver started at http://127.0.97.3:36159/ using document root <none> and password file <none>
I20260928 05:21:51.662156 18937 fs_manager.cc:359] Metadata directory not provided
I20260928 05:21:51.662281 18937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:21:51.662585 18937 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:21:51.665515 18937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572622576160-388-0/minicluster-data/ts-2/data/instance:
uuid: "af9419200157403297f42945af9613a6"
format_stamp: "Formatted at 2026-09-28 05:21:51 on dist-test-slave-whsn"
server_key: "9d49866e2ea78260b3ff368e55e7a3cc"
server_key_iv: "d9a1834d0179389602ba48486dea2d95"
server_key_version: "encryptionkey@0"
I20260928 05:21:51.666311 18937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572622576160-388-0/minicluster-data/ts-2/wal/instance:
uuid: "af9419200157403297f42945af9613a6"
format_stamp: "Formatted at 2026-09-28 05:21:51 on dist-test-slave-whsn"
server_key: "9d49866e2ea78260b3ff368e55e7a3cc"
server_key_iv: "d9a1834d0179389602ba48486dea2d95"
server_key_version: "encryptionkey@0"
I20260928 05:21:51.671423 18937 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260928 05:21:51.674984 18952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:21:51.676327 18937 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 05:21:51.676476 18937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572622576160-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572622576160-388-0/minicluster-data/ts-2/wal
uuid: "af9419200157403297f42945af9613a6"
format_stamp: "Formatted at 2026-09-28 05:21:51 on dist-test-slave-whsn"
server_key: "9d49866e2ea78260b3ff368e55e7a3cc"
server_key_iv: "d9a1834d0179389602ba48486dea2d95"
server_key_version: "encryptionkey@0"
I20260928 05:21:51.676644 18937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572622576160-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572622576160-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572622576160-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:21:51.694416 18937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:21:51.698110 18937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:21:51.698347 18937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:21:51.699193 18937 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:21:51.700439 18937 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:21:51.700515 18937 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:21:51.700629 18937 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:21:51.700680 18937 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:21:51.729683 18937 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44657
I20260928 05:21:51.729727 19064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44657 every 8 connection(s)
I20260928 05:21:51.730851 18937 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572622576160-388-0/minicluster-data/ts-2/data/info.pb
I20260928 05:21:51.733332 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 18937
I20260928 05:21:51.733532 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572622576160-388-0/minicluster-data/ts-2/wal/instance
I20260928 05:21:51.733909 388 external_mini_cluster.cc:1428] Setting key b763ac44048da84a99d51ca47fcd89e6
Sep 28 05:21:51 dist-test-slave-whsn krb5kdc[18564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572911, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:21:51.749246 18800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:51.733239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59237 (local address 127.0.97.62:44433)
0928 05:21:51.733441 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 05:21:51.733447 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:21:51.734699 (+ 1252us) server_negotiation.cc:407] Connection header received
0928 05:21:51.736683 (+ 1984us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:21:51.736687 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:21:51.736751 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:21:51.736823 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:21:51.738307 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:51.739047 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:51.740473 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:51.740912 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:51.741010 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:51.744043 (+ 3033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:21:51.744047 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:21:51.744050 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:21:51.744082 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:21:51.746404 (+ 2322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:21:51.747144 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:21:51.747150 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:21:51.747153 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:21:51.747275 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:21:51.747653 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:21:51.747658 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:21:51.747662 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:21:51.747936 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:21:51.748075 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 05:21:51.748681 (+ 606us) server_negotiation.cc:299] Negotiation successful
0928 05:21:51.748912 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 05:21:51.750075 19067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:51.733770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44433 (local address 127.0.97.3:59237)
0928 05:21:51.734543 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:51.734594 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 05:21:51.736395 (+ 1801us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:21:51.737044 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:21:51.737051 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:21:51.737588 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:21:51.738143 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:51.738161 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:51.739171 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:51.739175 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:51.740368 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:51.740379 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:51.741098 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:51.741103 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:51.741301 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:51.741866 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:21:51.741892 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:21:51.743855 (+ 1963us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:21:51.746554 (+ 2699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:21:51.746563 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:21:51.746576 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:21:51.747014 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:21:51.747376 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:21:51.747380 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:21:51.747382 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:21:51.747552 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:21:51.748054 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:21:51.748061 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:21:51.748351 (+ 290us) client_negotiation.cc:769] Sending connection context
0928 05:21:51.748582 (+ 231us) client_negotiation.cc:240] Negotiation successful
0928 05:21:51.749057 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":312,"threads_started":1}
I20260928 05:21:51.751227 19065 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44433
I20260928 05:21:51.751452 19065 heartbeater.cc:464] Registering TS with master...
I20260928 05:21:51.752061 19065 heartbeater.cc:511] Master 127.0.97.62:44433 requested a full tablet report, sending...
I20260928 05:21:51.753288 18609 ts_manager.cc:194] Registered new tserver with Master: af9419200157403297f42945af9613a6 (127.0.97.3:44657)
I20260928 05:21:51.754014 18609 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:59237
I20260928 05:21:51.760006 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 05:21:51.776634 18800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:51.763670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46184 (local address 127.0.97.62:44433)
0928 05:21:51.763817 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 05:21:51.763822 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:21:51.764454 (+ 632us) server_negotiation.cc:407] Connection header received
0928 05:21:51.764613 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:21:51.764616 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:21:51.764678 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:21:51.764749 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:21:51.766438 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:51.767181 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:51.769021 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:51.769409 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:51.769493 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:51.771847 (+ 2354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:21:51.771851 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:21:51.771855 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:21:51.771882 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:21:51.774307 (+ 2425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:21:51.774920 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:21:51.774924 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:21:51.774926 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:21:51.775046 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:21:51.775396 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:21:51.775399 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:21:51.775402 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:21:51.775676 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:21:51.775866 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0928 05:21:51.776239 (+ 373us) server_negotiation.cc:299] Negotiation successful
0928 05:21:51.776360 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 05:21:51.779960 18609 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46184:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 05:21:51.786653 18609 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 05:21:51.800683 18798 heartbeater.cc:503] Master 127.0.97.62:44433 was elected leader, sending a full tablet report...
I20260928 05:21:51.819921 19078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:51.798365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43083 (local address 127.0.0.1:48996)
0928 05:21:51.811584 (+ 13219us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:51.811603 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 05:21:51.811721 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:21:51.812664 (+ 943us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:21:51.812672 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:21:51.812703 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:21:51.813106 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:51.813119 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:51.814937 (+ 1818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:51.814943 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:51.817534 (+ 2591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:51.817546 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:51.819053 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:51.819059 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:51.819245 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:51.819300 (+ 55us) client_negotiation.cc:769] Sending connection context
0928 05:21:51.819416 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 05:21:51.819580 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13090,"thread_start_us":12968,"threads_started":1}
I20260928 05:21:51.819962 19080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:51.798617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48996 (local address 127.0.97.2:43083)
0928 05:21:51.811656 (+ 13039us) server_negotiation.cc:206] Beginning negotiation
0928 05:21:51.811665 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:21:51.811693 (+ 28us) server_negotiation.cc:407] Connection header received
0928 05:21:51.812150 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:21:51.812159 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:21:51.812426 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:21:51.812632 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:21:51.813256 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:51.814774 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:51.817689 (+ 2915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:51.818854 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:51.818980 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:51.819186 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0928 05:21:51.819486 (+ 300us) server_negotiation.cc:299] Negotiation successful
0928 05:21:51.819645 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12886,"thread_start_us":13196,"threads_started":1}
I20260928 05:21:51.821442 19077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:51.798230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35993 (local address 127.0.0.1:35480)
0928 05:21:51.810892 (+ 12662us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:51.810942 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 05:21:51.811116 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:21:51.814792 (+ 3676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:21:51.814796 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:21:51.814821 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:21:51.815276 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:51.815285 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:51.817167 (+ 1882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:51.817172 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:51.818928 (+ 1756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:51.818940 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:51.820688 (+ 1748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:51.820695 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:51.820869 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:51.820905 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 05:21:51.820973 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 05:21:51.821090 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12393,"thread_start_us":13464,"threads_started":1}
I20260928 05:21:51.822196 19079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:51.798391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35480 (local address 127.0.97.1:35993)
0928 05:21:51.814271 (+ 15880us) server_negotiation.cc:206] Beginning negotiation
0928 05:21:51.814281 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 05:21:51.814308 (+ 27us) server_negotiation.cc:407] Connection header received
0928 05:21:51.814388 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:21:51.814395 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:21:51.814619 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:21:51.814801 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:21:51.815473 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:51.817021 (+ 1548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:51.819099 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:51.820446 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:51.821537 (+ 1091us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:51.821761 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0928 05:21:51.821825 (+ 64us) server_negotiation.cc:299] Negotiation successful
0928 05:21:51.821983 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15718,"thread_start_us":16572,"threads_started":1}
I20260928 05:21:51.823812 18867 tablet_service.cc:1467] Processing CreateTablet for tablet 2710549fb0744f16b725676c384535b3 (DEFAULT_TABLE table=test-table [id=bbef5e23755e40e5a4630380246344f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:21:51.825124 18867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2710549fb0744f16b725676c384535b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:21:51.825811 19081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:51.800089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44657 (local address 127.0.0.1:52466)
0928 05:21:51.817179 (+ 17090us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:51.817194 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 05:21:51.817305 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:21:51.818017 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:21:51.818022 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:21:51.818050 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:21:51.818401 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:51.818411 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:51.820121 (+ 1710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:51.820126 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:51.823117 (+ 2991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:51.823131 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:51.825139 (+ 2008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:51.825146 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:51.825302 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:51.825333 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 05:21:51.825418 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 05:21:51.825533 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16957,"thread_start_us":16811,"threads_started":1}
I20260928 05:21:51.825505 18733 tablet_service.cc:1467] Processing CreateTablet for tablet 2710549fb0744f16b725676c384535b3 (DEFAULT_TABLE table=test-table [id=bbef5e23755e40e5a4630380246344f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:21:51.826069 19082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:51.800718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52466 (local address 127.0.97.3:44657)
0928 05:21:51.816735 (+ 16017us) server_negotiation.cc:206] Beginning negotiation
0928 05:21:51.816743 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:21:51.817431 (+ 688us) server_negotiation.cc:407] Connection header received
0928 05:21:51.817538 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:21:51.817544 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:21:51.817750 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:21:51.817905 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:21:51.818594 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:51.819968 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:51.823278 (+ 3310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:51.824617 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:51.824739 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:51.824999 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0928 05:21:51.825623 (+ 624us) server_negotiation.cc:299] Negotiation successful
0928 05:21:51.825777 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15857,"thread_start_us":15702,"threads_started":1}
I20260928 05:21:51.826856 18733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2710549fb0744f16b725676c384535b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:21:51.829303 19000 tablet_service.cc:1467] Processing CreateTablet for tablet 2710549fb0744f16b725676c384535b3 (DEFAULT_TABLE table=test-table [id=bbef5e23755e40e5a4630380246344f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:21:51.830642 19000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2710549fb0744f16b725676c384535b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:21:51.837586 19083 tablet_bootstrap.cc:492] T 2710549fb0744f16b725676c384535b3 P 9610853da7d24baea0389d4316e3086b: Bootstrap starting.
I20260928 05:21:51.839351 19084 tablet_bootstrap.cc:492] T 2710549fb0744f16b725676c384535b3 P 401cfcf03bd54a4f90027259e5c5ec38: Bootstrap starting.
I20260928 05:21:51.841077 19083 tablet_bootstrap.cc:654] T 2710549fb0744f16b725676c384535b3 P 9610853da7d24baea0389d4316e3086b: Neither blocks nor log segments found. Creating new log.
I20260928 05:21:51.842608 19084 tablet_bootstrap.cc:654] T 2710549fb0744f16b725676c384535b3 P 401cfcf03bd54a4f90027259e5c5ec38: Neither blocks nor log segments found. Creating new log.
I20260928 05:21:51.842774 19085 tablet_bootstrap.cc:492] T 2710549fb0744f16b725676c384535b3 P af9419200157403297f42945af9613a6: Bootstrap starting.
I20260928 05:21:51.843739 19084 log.cc:824] T 2710549fb0744f16b725676c384535b3 P 401cfcf03bd54a4f90027259e5c5ec38: Log is configured to *not* fsync() on all Append() calls
I20260928 05:21:51.843349 19083 log.cc:824] T 2710549fb0744f16b725676c384535b3 P 9610853da7d24baea0389d4316e3086b: Log is configured to *not* fsync() on all Append() calls
I20260928 05:21:51.845942 19085 tablet_bootstrap.cc:654] T 2710549fb0744f16b725676c384535b3 P af9419200157403297f42945af9613a6: Neither blocks nor log segments found. Creating new log.
I20260928 05:21:51.846267 19084 tablet_bootstrap.cc:492] T 2710549fb0744f16b725676c384535b3 P 401cfcf03bd54a4f90027259e5c5ec38: No bootstrap required, opened a new log
I20260928 05:21:51.846484 19084 ts_tablet_manager.cc:1397] T 2710549fb0744f16b725676c384535b3 P 401cfcf03bd54a4f90027259e5c5ec38: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260928 05:21:51.847019 19083 tablet_bootstrap.cc:492] T 2710549fb0744f16b725676c384535b3 P 9610853da7d24baea0389d4316e3086b: No bootstrap required, opened a new log
I20260928 05:21:51.847091 19085 log.cc:824] T 2710549fb0744f16b725676c384535b3 P af9419200157403297f42945af9613a6: Log is configured to *not* fsync() on all Append() calls
I20260928 05:21:51.847263 19083 ts_tablet_manager.cc:1397] T 2710549fb0744f16b725676c384535b3 P 9610853da7d24baea0389d4316e3086b: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260928 05:21:51.849757 19085 tablet_bootstrap.cc:492] T 2710549fb0744f16b725676c384535b3 P af9419200157403297f42945af9613a6: No bootstrap required, opened a new log
I20260928 05:21:51.849985 19085 ts_tablet_manager.cc:1397] T 2710549fb0744f16b725676c384535b3 P af9419200157403297f42945af9613a6: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.001s
I20260928 05:21:51.853854 19084 raft_consensus.cc:348] T 2710549fb0744f16b725676c384535b3 P 401cfcf03bd54a4f90027259e5c5ec38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9610853da7d24baea0389d4316e3086b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35993 } } peers { permanent_uuid: "401cfcf03bd54a4f90027259e5c5ec38" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43083 } } peers { permanent_uuid: "af9419200157403297f42945af9613a6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44657 } }
I20260928 05:21:51.854218 19084 raft_consensus.cc:374] T 2710549fb0744f16b725676c384535b3 P 401cfcf03bd54a4f90027259e5c5ec38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:21:51.854338 19084 raft_consensus.cc:729] T 2710549fb0744f16b725676c384535b3 P 401cfcf03bd54a4f90027259e5c5ec38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 401cfcf03bd54a4f90027259e5c5ec38, State: Initialized, Role: FOLLOWER
I20260928 05:21:51.854871 19084 consensus_queue.cc:260] T 2710549fb0744f16b725676c384535b3 P 401cfcf03bd54a4f90027259e5c5ec38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9610853da7d24baea0389d4316e3086b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35993 } } peers { permanent_uuid: "401cfcf03bd54a4f90027259e5c5ec38" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43083 } } peers { permanent_uuid: "af9419200157403297f42945af9613a6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44657 } }
I20260928 05:21:51.855877 18932 heartbeater.cc:503] Master 127.0.97.62:44433 was elected leader, sending a full tablet report...
I20260928 05:21:51.856482 19084 ts_tablet_manager.cc:1428] T 2710549fb0744f16b725676c384535b3 P 401cfcf03bd54a4f90027259e5c5ec38: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260928 05:21:51.857052 19083 raft_consensus.cc:348] T 2710549fb0744f16b725676c384535b3 P 9610853da7d24baea0389d4316e3086b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9610853da7d24baea0389d4316e3086b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35993 } } peers { permanent_uuid: "401cfcf03bd54a4f90027259e5c5ec38" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43083 } } peers { permanent_uuid: "af9419200157403297f42945af9613a6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44657 } }
I20260928 05:21:51.857443 19083 raft_consensus.cc:374] T 2710549fb0744f16b725676c384535b3 P 9610853da7d24baea0389d4316e3086b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:21:51.857544 19083 raft_consensus.cc:729] T 2710549fb0744f16b725676c384535b3 P 9610853da7d24baea0389d4316e3086b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9610853da7d24baea0389d4316e3086b, State: Initialized, Role: FOLLOWER
I20260928 05:21:51.858105 19083 consensus_queue.cc:260] T 2710549fb0744f16b725676c384535b3 P 9610853da7d24baea0389d4316e3086b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9610853da7d24baea0389d4316e3086b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35993 } } peers { permanent_uuid: "401cfcf03bd54a4f90027259e5c5ec38" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43083 } } peers { permanent_uuid: "af9419200157403297f42945af9613a6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44657 } }
I20260928 05:21:51.860127 19085 raft_consensus.cc:348] T 2710549fb0744f16b725676c384535b3 P af9419200157403297f42945af9613a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9610853da7d24baea0389d4316e3086b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35993 } } peers { permanent_uuid: "401cfcf03bd54a4f90027259e5c5ec38" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43083 } } peers { permanent_uuid: "af9419200157403297f42945af9613a6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44657 } }
I20260928 05:21:51.860543 19085 raft_consensus.cc:374] T 2710549fb0744f16b725676c384535b3 P af9419200157403297f42945af9613a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:21:51.860634 19085 raft_consensus.cc:729] T 2710549fb0744f16b725676c384535b3 P af9419200157403297f42945af9613a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af9419200157403297f42945af9613a6, State: Initialized, Role: FOLLOWER
I20260928 05:21:51.861251 19085 consensus_queue.cc:260] T 2710549fb0744f16b725676c384535b3 P af9419200157403297f42945af9613a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9610853da7d24baea0389d4316e3086b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35993 } } peers { permanent_uuid: "401cfcf03bd54a4f90027259e5c5ec38" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43083 } } peers { permanent_uuid: "af9419200157403297f42945af9613a6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44657 } }
I20260928 05:21:51.862507 19065 heartbeater.cc:503] Master 127.0.97.62:44433 was elected leader, sending a full tablet report...
I20260928 05:21:51.863061 19085 ts_tablet_manager.cc:1428] T 2710549fb0744f16b725676c384535b3 P af9419200157403297f42945af9613a6: Time spent starting tablet: real 0.013s user 0.006s sys 0.007s
I20260928 05:21:51.864504 19083 ts_tablet_manager.cc:1428] T 2710549fb0744f16b725676c384535b3 P 9610853da7d24baea0389d4316e3086b: Time spent starting tablet: real 0.017s user 0.014s sys 0.004s
W20260928 05:21:51.982360 19066 tablet.cc:2369] T 2710549fb0744f16b725676c384535b3 P af9419200157403297f42945af9613a6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 05:21:52.011823 18933 tablet.cc:2369] T 2710549fb0744f16b725676c384535b3 P 401cfcf03bd54a4f90027259e5c5ec38: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 05:21:52.017483 18799 tablet.cc:2369] T 2710549fb0744f16b725676c384535b3 P 9610853da7d24baea0389d4316e3086b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 05:21:52.042541 19089 raft_consensus.cc:482] T 2710549fb0744f16b725676c384535b3 P 401cfcf03bd54a4f90027259e5c5ec38 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:21:52.042809 19089 raft_consensus.cc:504] T 2710549fb0744f16b725676c384535b3 P 401cfcf03bd54a4f90027259e5c5ec38 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9610853da7d24baea0389d4316e3086b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35993 } } peers { permanent_uuid: "401cfcf03bd54a4f90027259e5c5ec38" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43083 } } peers { permanent_uuid: "af9419200157403297f42945af9613a6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44657 } }
I20260928 05:21:52.044445 19089 leader_election.cc:288] T 2710549fb0744f16b725676c384535b3 P 401cfcf03bd54a4f90027259e5c5ec38 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9610853da7d24baea0389d4316e3086b (127.0.97.1:35993), af9419200157403297f42945af9613a6 (127.0.97.3:44657)
I20260928 05:21:52.051656 19082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:52.044835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48173 (local address 127.0.97.3:44657)
0928 05:21:52.045075 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0928 05:21:52.045082 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:21:52.045916 (+ 834us) server_negotiation.cc:407] Connection header received
0928 05:21:52.046062 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:21:52.046067 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:21:52.046139 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:21:52.046229 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:21:52.046821 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:52.047564 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:52.050002 (+ 2438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:52.050752 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:52.050874 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:52.050969 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 05:21:52.051303 (+ 334us) server_negotiation.cc:299] Negotiation successful
0928 05:21:52.051423 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260928 05:21:52.051654 19079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:52.044999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59239 (local address 127.0.97.1:35993)
0928 05:21:52.045212 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 05:21:52.045218 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:21:52.045718 (+ 500us) server_negotiation.cc:407] Connection header received
0928 05:21:52.045895 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:21:52.045900 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:21:52.045972 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:21:52.046058 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:21:52.046627 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:52.047397 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:52.049951 (+ 2554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:52.050714 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:52.050817 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:52.050916 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 05:21:52.051270 (+ 354us) server_negotiation.cc:299] Negotiation successful
0928 05:21:52.051396 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 05:21:52.051806 19093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:52.044908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44657 (local address 127.0.97.2:48173)
0928 05:21:52.045810 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:52.045826 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 05:21:52.045947 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:21:52.046253 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:21:52.046256 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:21:52.046278 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:21:52.046667 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:52.046679 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:52.047697 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:52.047705 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:52.049811 (+ 2106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:52.049825 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:52.050975 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:52.050981 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:52.051159 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:52.051182 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 05:21:52.051267 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 05:21:52.051388 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":791,"mutex_wait_us":72,"thread_start_us":504,"threads_started":1}
I20260928 05:21:52.051846 19092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:52.044761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35993 (local address 127.0.97.2:59239)
0928 05:21:52.045627 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:52.045646 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 05:21:52.045767 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:21:52.046090 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:21:52.046094 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:21:52.046116 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:21:52.046480 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:52.046489 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:52.047526 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:52.047532 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:52.049813 (+ 2281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:52.049825 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:52.050942 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:52.050949 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:52.051092 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:52.051147 (+ 55us) client_negotiation.cc:769] Sending connection context
0928 05:21:52.051217 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 05:21:52.051333 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":743,"mutex_wait_us":43,"thread_start_us":485,"threads_started":1}
I20260928 05:21:52.052660 19020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2710549fb0744f16b725676c384535b3" candidate_uuid: "401cfcf03bd54a4f90027259e5c5ec38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af9419200157403297f42945af9613a6" is_pre_election: true
I20260928 05:21:52.052889 18753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2710549fb0744f16b725676c384535b3" candidate_uuid: "401cfcf03bd54a4f90027259e5c5ec38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9610853da7d24baea0389d4316e3086b" is_pre_election: true
I20260928 05:21:52.053156 19020 raft_consensus.cc:2454] T 2710549fb0744f16b725676c384535b3 P af9419200157403297f42945af9613a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 401cfcf03bd54a4f90027259e5c5ec38 in term 0.
I20260928 05:21:52.053349 18753 raft_consensus.cc:2454] T 2710549fb0744f16b725676c384535b3 P 9610853da7d24baea0389d4316e3086b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 401cfcf03bd54a4f90027259e5c5ec38 in term 0.
I20260928 05:21:52.053728 18823 leader_election.cc:302] T 2710549fb0744f16b725676c384535b3 P 401cfcf03bd54a4f90027259e5c5ec38 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 401cfcf03bd54a4f90027259e5c5ec38, af9419200157403297f42945af9613a6; no voters:
I20260928 05:21:52.054133 19089 raft_consensus.cc:2781] T 2710549fb0744f16b725676c384535b3 P 401cfcf03bd54a4f90027259e5c5ec38 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 05:21:52.054260 19089 raft_consensus.cc:482] T 2710549fb0744f16b725676c384535b3 P 401cfcf03bd54a4f90027259e5c5ec38 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 05:21:52.054350 19089 raft_consensus.cc:3037] T 2710549fb0744f16b725676c384535b3 P 401cfcf03bd54a4f90027259e5c5ec38 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:21:52.056403 19089 raft_consensus.cc:504] T 2710549fb0744f16b725676c384535b3 P 401cfcf03bd54a4f90027259e5c5ec38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9610853da7d24baea0389d4316e3086b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35993 } } peers { permanent_uuid: "401cfcf03bd54a4f90027259e5c5ec38" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43083 } } peers { permanent_uuid: "af9419200157403297f42945af9613a6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44657 } }
I20260928 05:21:52.057022 19089 leader_election.cc:288] T 2710549fb0744f16b725676c384535b3 P 401cfcf03bd54a4f90027259e5c5ec38 [CANDIDATE]: Term 1 election: Requested vote from peers 9610853da7d24baea0389d4316e3086b (127.0.97.1:35993), af9419200157403297f42945af9613a6 (127.0.97.3:44657)
I20260928 05:21:52.057382 18753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2710549fb0744f16b725676c384535b3" candidate_uuid: "401cfcf03bd54a4f90027259e5c5ec38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9610853da7d24baea0389d4316e3086b"
I20260928 05:21:52.057482 19020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2710549fb0744f16b725676c384535b3" candidate_uuid: "401cfcf03bd54a4f90027259e5c5ec38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af9419200157403297f42945af9613a6"
I20260928 05:21:52.057569 18753 raft_consensus.cc:3037] T 2710549fb0744f16b725676c384535b3 P 9610853da7d24baea0389d4316e3086b [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:21:52.057708 19020 raft_consensus.cc:3037] T 2710549fb0744f16b725676c384535b3 P af9419200157403297f42945af9613a6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:21:52.059799 18753 raft_consensus.cc:2454] T 2710549fb0744f16b725676c384535b3 P 9610853da7d24baea0389d4316e3086b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 401cfcf03bd54a4f90027259e5c5ec38 in term 1.
I20260928 05:21:52.059921 19020 raft_consensus.cc:2454] T 2710549fb0744f16b725676c384535b3 P af9419200157403297f42945af9613a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 401cfcf03bd54a4f90027259e5c5ec38 in term 1.
I20260928 05:21:52.060138 18822 leader_election.cc:302] T 2710549fb0744f16b725676c384535b3 P 401cfcf03bd54a4f90027259e5c5ec38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 401cfcf03bd54a4f90027259e5c5ec38, 9610853da7d24baea0389d4316e3086b; no voters:
I20260928 05:21:52.060443 19089 raft_consensus.cc:2781] T 2710549fb0744f16b725676c384535b3 P 401cfcf03bd54a4f90027259e5c5ec38 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:21:52.060877 19089 raft_consensus.cc:686] T 2710549fb0744f16b725676c384535b3 P 401cfcf03bd54a4f90027259e5c5ec38 [term 1 LEADER]: Becoming Leader. State: Replica: 401cfcf03bd54a4f90027259e5c5ec38, State: Running, Role: LEADER
I20260928 05:21:52.061383 19089 consensus_queue.cc:237] T 2710549fb0744f16b725676c384535b3 P 401cfcf03bd54a4f90027259e5c5ec38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9610853da7d24baea0389d4316e3086b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35993 } } peers { permanent_uuid: "401cfcf03bd54a4f90027259e5c5ec38" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43083 } } peers { permanent_uuid: "af9419200157403297f42945af9613a6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44657 } }
I20260928 05:21:52.065722 18609 catalog_manager.cc:5495] T 2710549fb0744f16b725676c384535b3 P 401cfcf03bd54a4f90027259e5c5ec38 reported cstate change: term changed from 0 to 1, leader changed from <none> to 401cfcf03bd54a4f90027259e5c5ec38 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "401cfcf03bd54a4f90027259e5c5ec38" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9610853da7d24baea0389d4316e3086b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "401cfcf03bd54a4f90027259e5c5ec38" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af9419200157403297f42945af9613a6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44657 } health_report { overall_health: UNKNOWN } } }
I20260928 05:21:52.116387 19080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:52.111544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49006 (local address 127.0.97.2:43083)
0928 05:21:52.111712 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 05:21:52.111716 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:21:52.111733 (+ 17us) server_negotiation.cc:407] Connection header received
0928 05:21:52.111808 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:21:52.111811 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:21:52.111895 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:21:52.112013 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:21:52.112341 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:52.113064 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:52.114420 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:52.114801 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:52.114892 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:52.115338 (+ 446us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:21:52.115818 (+ 480us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:21:52.115935 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 05:21:52.116142 (+ 207us) server_negotiation.cc:299] Negotiation successful
0928 05:21:52.116179 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 05:21:52.124248 19020 raft_consensus.cc:1261] T 2710549fb0744f16b725676c384535b3 P af9419200157403297f42945af9613a6 [term 1 FOLLOWER]: Refusing update from remote peer 401cfcf03bd54a4f90027259e5c5ec38: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:21:52.124248 18753 raft_consensus.cc:1261] T 2710549fb0744f16b725676c384535b3 P 9610853da7d24baea0389d4316e3086b [term 1 FOLLOWER]: Refusing update from remote peer 401cfcf03bd54a4f90027259e5c5ec38: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:21:52.125322 19094 consensus_queue.cc:1035] T 2710549fb0744f16b725676c384535b3 P 401cfcf03bd54a4f90027259e5c5ec38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af9419200157403297f42945af9613a6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:21:52.125626 19089 consensus_queue.cc:1035] T 2710549fb0744f16b725676c384535b3 P 401cfcf03bd54a4f90027259e5c5ec38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9610853da7d24baea0389d4316e3086b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:21:52.148466 18608 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46184:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:21:52.148702 18608 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46184:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:21:52.152123 18608 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 549e378ef8a743fea5eeec8560f320ff: Sending DeleteTablet for 3 replicas of tablet 2710549fb0744f16b725676c384535b3
I20260928 05:21:52.153146 18733 tablet_service.cc:1514] Processing DeleteTablet for tablet 2710549fb0744f16b725676c384535b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:21:52 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35480
I20260928 05:21:52.153175 18867 tablet_service.cc:1514] Processing DeleteTablet for tablet 2710549fb0744f16b725676c384535b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:21:52 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48996
I20260928 05:21:52.153715 19000 tablet_service.cc:1514] Processing DeleteTablet for tablet 2710549fb0744f16b725676c384535b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:21:52 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52466
I20260928 05:21:52.157861 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 18670
I20260928 05:21:52.169775 19107 tablet_replica.cc:330] T 2710549fb0744f16b725676c384535b3 P 401cfcf03bd54a4f90027259e5c5ec38: stopping tablet replica
I20260928 05:21:52.170315 19107 raft_consensus.cc:2229] T 2710549fb0744f16b725676c384535b3 P 401cfcf03bd54a4f90027259e5c5ec38 [term 1 LEADER]: Raft consensus shutting down.
I20260928 05:21:52.171092 19107 raft_consensus.cc:2258] T 2710549fb0744f16b725676c384535b3 P 401cfcf03bd54a4f90027259e5c5ec38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:21:52.173334 19107 ts_tablet_manager.cc:1905] T 2710549fb0744f16b725676c384535b3 P 401cfcf03bd54a4f90027259e5c5ec38: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:21:52.180822 19107 ts_tablet_manager.cc:1918] T 2710549fb0744f16b725676c384535b3 P 401cfcf03bd54a4f90027259e5c5ec38: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:21:52.181082 19107 log.cc:1196] T 2710549fb0744f16b725676c384535b3 P 401cfcf03bd54a4f90027259e5c5ec38: Deleting WAL directory at /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572622576160-388-0/minicluster-data/ts-1/wal/wals/2710549fb0744f16b725676c384535b3
I20260928 05:21:52.181701 19107 ts_tablet_manager.cc:1939] T 2710549fb0744f16b725676c384535b3 P 401cfcf03bd54a4f90027259e5c5ec38: Deleting consensus metadata
I20260928 05:21:52.183506 18593 catalog_manager.cc:4841] TS 401cfcf03bd54a4f90027259e5c5ec38 (127.0.97.2:43083): tablet 2710549fb0744f16b725676c384535b3 (table test-table [id=bbef5e23755e40e5a4630380246344f6]) successfully deleted
I20260928 05:21:52.185315 19108 tablet_replica.cc:330] T 2710549fb0744f16b725676c384535b3 P af9419200157403297f42945af9613a6: stopping tablet replica
I20260928 05:21:52.185825 19108 raft_consensus.cc:2229] T 2710549fb0744f16b725676c384535b3 P af9419200157403297f42945af9613a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:21:52.186383 19108 raft_consensus.cc:2258] T 2710549fb0744f16b725676c384535b3 P af9419200157403297f42945af9613a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:21:52.188587 19108 ts_tablet_manager.cc:1905] T 2710549fb0744f16b725676c384535b3 P af9419200157403297f42945af9613a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:21:52.191007 19106 tablet_replica.cc:330] T 2710549fb0744f16b725676c384535b3 P 9610853da7d24baea0389d4316e3086b: stopping tablet replica
I20260928 05:21:52.191406 19106 raft_consensus.cc:2229] T 2710549fb0744f16b725676c384535b3 P 9610853da7d24baea0389d4316e3086b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:21:52.191929 19106 raft_consensus.cc:2258] T 2710549fb0744f16b725676c384535b3 P 9610853da7d24baea0389d4316e3086b [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 28 05:21:52 dist-test-slave-whsn krb5kdc[18564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572909, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260928 05:21:52.195770 19108 ts_tablet_manager.cc:1918] T 2710549fb0744f16b725676c384535b3 P af9419200157403297f42945af9613a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:21:52.196172 19108 log.cc:1196] T 2710549fb0744f16b725676c384535b3 P af9419200157403297f42945af9613a6: Deleting WAL directory at /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572622576160-388-0/minicluster-data/ts-2/wal/wals/2710549fb0744f16b725676c384535b3
I20260928 05:21:52.196707 19108 ts_tablet_manager.cc:1939] T 2710549fb0744f16b725676c384535b3 P af9419200157403297f42945af9613a6: Deleting consensus metadata
I20260928 05:21:52.198791 18596 catalog_manager.cc:4841] TS af9419200157403297f42945af9613a6 (127.0.97.3:44657): tablet 2710549fb0744f16b725676c384535b3 (table test-table [id=bbef5e23755e40e5a4630380246344f6]) successfully deleted
I20260928 05:21:52.198848 19106 ts_tablet_manager.cc:1905] T 2710549fb0744f16b725676c384535b3 P 9610853da7d24baea0389d4316e3086b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:21:52.201951 19079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:52.159320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35496 (local address 127.0.97.1:35993)
0928 05:21:52.159491 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 05:21:52.159497 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:21:52.183105 (+ 23608us) server_negotiation.cc:407] Connection header received
0928 05:21:52.183626 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:21:52.183633 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:21:52.183727 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:21:52.183800 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:21:52.185665 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:52.186919 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:52.188956 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:52.189664 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:52.189761 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:52.195223 (+ 5462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:21:52.195236 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:21:52.195252 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:21:52.195290 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:21:52.198630 (+ 3340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:21:52.199405 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:21:52.199413 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:21:52.199418 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:21:52.199590 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:21:52.200003 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:21:52.200008 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:21:52.200011 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:21:52.200459 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:21:52.200757 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0928 05:21:52.201343 (+ 586us) server_negotiation.cc:299] Negotiation successful
0928 05:21:52.201567 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 05:21:52.203218 18793 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:21:52.297662 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 18670
W20260928 05:21:52.317497 18595 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:35993 (error 108)
I20260928 05:21:52.317705 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 18804
I20260928 05:21:52.319571 19081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:52.319114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35993 (local address 127.0.0.1:35506)
0928 05:21:52.319295 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:52.319406 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260928 05:21:52.319922 18595 catalog_manager.cc:4584] TS 9610853da7d24baea0389d4316e3086b (127.0.97.1:35993): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2710549fb0744f16b725676c384535b3: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35993: connect: Connection refused (error 111)
Sep 28 05:21:52 dist-test-slave-whsn krb5kdc[18564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572909, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260928 05:21:52.332747 19080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:52.318788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49010 (local address 127.0.97.2:43083)
0928 05:21:52.318989 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 05:21:52.318995 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:21:52.319012 (+ 17us) server_negotiation.cc:407] Connection header received
0928 05:21:52.319075 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:21:52.319080 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:21:52.319155 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:21:52.319248 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:21:52.320679 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:52.321523 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:52.322865 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:52.323271 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:52.323377 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:52.327337 (+ 3960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:21:52.327348 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:21:52.327366 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:21:52.327397 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:21:52.330192 (+ 2795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:21:52.330780 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:21:52.330787 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:21:52.330789 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:21:52.330919 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:21:52.331266 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:21:52.331269 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:21:52.331271 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:21:52.331635 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:21:52.331966 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0928 05:21:52.332295 (+ 329us) server_negotiation.cc:299] Negotiation successful
0928 05:21:52.332459 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 05:21:52.333477 18927 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:21:52.373400 19081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:52.373005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35993 (local address 127.0.0.1:35518)
0928 05:21:52.373183 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:52.373267 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260928 05:21:52.374356 19081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:52.374029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35993 (local address 127.0.0.1:35520)
0928 05:21:52.374165 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:52.374228 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260928 05:21:52.434712 19081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:52.434168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35993 (local address 127.0.0.1:35528)
0928 05:21:52.434389 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:52.434506 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260928 05:21:52.435659 19081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:52.435316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35993 (local address 127.0.0.1:35536)
0928 05:21:52.435446 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:52.435533 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 05:21:52.443086 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 18804
I20260928 05:21:52.457396 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 18937
Sep 28 05:21:52 dist-test-slave-whsn krb5kdc[18564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572909, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260928 05:21:52.471292 19082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:52.458090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52474 (local address 127.0.97.3:44657)
0928 05:21:52.458256 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 05:21:52.458262 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:21:52.458281 (+ 19us) server_negotiation.cc:407] Connection header received
0928 05:21:52.458461 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:21:52.458467 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:21:52.458537 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:21:52.458618 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:21:52.459899 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:52.460591 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:52.461843 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:52.462233 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:52.462345 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:52.466369 (+ 4024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:21:52.466383 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:21:52.466398 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:21:52.466439 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:21:52.468763 (+ 2324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:21:52.469390 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:21:52.469398 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:21:52.469400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:21:52.469534 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:21:52.469863 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:21:52.469867 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:21:52.469869 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:21:52.470213 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:21:52.470477 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0928 05:21:52.470792 (+ 315us) server_negotiation.cc:299] Negotiation successful
0928 05:21:52.471004 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 05:21:52.471920 19060 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:21:52.516362 19081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:52.515782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35993 (local address 127.0.0.1:35548)
0928 05:21:52.516022 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:52.516135 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260928 05:21:52.517449 19081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:52.517032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35993 (local address 127.0.0.1:35558)
0928 05:21:52.517233 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:52.517315 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260928 05:21:52.561090 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 18937
I20260928 05:21:52.574285 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 18577
I20260928 05:21:52.575251 18639 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:21:52.682832 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 18577
2026-09-28T05:21:52Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4102 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-28T05:21:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:21:52Z Disabled control of system clock
I20260928 05:21:52.747920 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572622576160-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572622576160-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572622576160-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46781
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:41131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46781
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 05:21:53.051697 19118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:21:53.051949 19118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:21:53.052034 19118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:21:53.058894 19118 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:21:53.058985 19118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:21:53.059021 19118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:21:53.059075 19118 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:21:53.059116 19118 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:21:53.067726 19118 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572622576160-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572622576160-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46781
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572622576160-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:21:53.068833 19118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:21:53.070511 19118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:21:53.081350 19124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:21:53.081314 19126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:21:53.081383 19123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:21:53.082158 19118 server_base.cc:1034] running on GCE node
I20260928 05:21:53.082863 19118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:21:53.084120 19118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:21:53.085335 19118 hybrid_clock.cc:648] HybridClock initialized: now 1790572913085288 us; error 29 us; skew 500 ppm
I20260928 05:21:53.085701 19118 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 05:21:53.088097 19118 webserver.cc:460] Webserver started at http://127.0.97.62:34843/ using document root <none> and password file <none>
I20260928 05:21:53.088714 19118 fs_manager.cc:359] Metadata directory not provided
I20260928 05:21:53.088799 19118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:21:53.089128 19118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:21:53.091789 19118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572622576160-388-0/minicluster-data/master-0/data/instance:
uuid: "34f88604b2924a1a982b30e5a85c581d"
format_stamp: "Formatted at 2026-09-28 05:21:53 on dist-test-slave-whsn"
server_key: "8499a4a10f89251411a73a7b237f21a8"
server_key_iv: "61ce1066913981330ad096ee13008b7b"
server_key_version: "encryptionkey@0"
I20260928 05:21:53.092515 19118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572622576160-388-0/minicluster-data/master-0/wal/instance:
uuid: "34f88604b2924a1a982b30e5a85c581d"
format_stamp: "Formatted at 2026-09-28 05:21:53 on dist-test-slave-whsn"
server_key: "8499a4a10f89251411a73a7b237f21a8"
server_key_iv: "61ce1066913981330ad096ee13008b7b"
server_key_version: "encryptionkey@0"
I20260928 05:21:53.097191 19118 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260928 05:21:53.100255 19132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:21:53.101490 19118 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 05:21:53.101639 19118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572622576160-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572622576160-388-0/minicluster-data/master-0/wal
uuid: "34f88604b2924a1a982b30e5a85c581d"
format_stamp: "Formatted at 2026-09-28 05:21:53 on dist-test-slave-whsn"
server_key: "8499a4a10f89251411a73a7b237f21a8"
server_key_iv: "61ce1066913981330ad096ee13008b7b"
server_key_version: "encryptionkey@0"
I20260928 05:21:53.101814 19118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572622576160-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572622576160-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572622576160-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:21:53.120158 19118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:21:53.120987 19118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:21:53.121239 19118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:21:53.137768 19118 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46781
I20260928 05:21:53.137792 19183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46781 every 8 connection(s)
I20260928 05:21:53.138948 19118 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:21:53.142818 19184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:21:53.145773 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 19118
I20260928 05:21:53.145902 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572622576160-388-0/minicluster-data/master-0/wal/instance
I20260928 05:21:53.146334 388 external_mini_cluster.cc:1428] Setting key aeb38e8b25a30f3e3b8d105109550b82
I20260928 05:21:53.154441 19184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34f88604b2924a1a982b30e5a85c581d: Bootstrap starting.
I20260928 05:21:53.157747 19184 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 34f88604b2924a1a982b30e5a85c581d: Neither blocks nor log segments found. Creating new log.
I20260928 05:21:53.158805 19184 log.cc:824] T 00000000000000000000000000000000 P 34f88604b2924a1a982b30e5a85c581d: Log is configured to *not* fsync() on all Append() calls
I20260928 05:21:53.159211 19187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:53.148189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40150 (local address 127.0.97.62:46781)
0928 05:21:53.149097 (+ 908us) server_negotiation.cc:206] Beginning negotiation
0928 05:21:53.149107 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 05:21:53.149146 (+ 39us) server_negotiation.cc:407] Connection header received
0928 05:21:53.150755 (+ 1609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:21:53.150776 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:21:53.151243 (+ 467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:21:53.151556 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:21:53.152144 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:53.153413 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:53.155180 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:53.155814 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:53.155938 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:53.157107 (+ 1169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:21:53.157114 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:21:53.157127 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:21:53.157608 (+ 481us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:21:53.157694 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:21:53.157714 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:21:53.157807 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 05:21:53.158028 (+ 221us) server_negotiation.cc:299] Negotiation successful
0928 05:21:53.158309 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":302,"threads_started":1}
I20260928 05:21:53.161302 19184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34f88604b2924a1a982b30e5a85c581d: No bootstrap required, opened a new log
I20260928 05:21:53.167934 19184 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34f88604b2924a1a982b30e5a85c581d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34f88604b2924a1a982b30e5a85c581d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46781 } }
I20260928 05:21:53.168251 19184 raft_consensus.cc:374] T 00000000000000000000000000000000 P 34f88604b2924a1a982b30e5a85c581d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:21:53.168339 19184 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34f88604b2924a1a982b30e5a85c581d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34f88604b2924a1a982b30e5a85c581d, State: Initialized, Role: FOLLOWER
I20260928 05:21:53.168881 19184 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34f88604b2924a1a982b30e5a85c581d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34f88604b2924a1a982b30e5a85c581d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46781 } }
I20260928 05:21:53.169106 19184 raft_consensus.cc:388] T 00000000000000000000000000000000 P 34f88604b2924a1a982b30e5a85c581d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:21:53.169230 19184 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34f88604b2924a1a982b30e5a85c581d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:21:53.169373 19184 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34f88604b2924a1a982b30e5a85c581d [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:21:53.171361 19184 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34f88604b2924a1a982b30e5a85c581d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34f88604b2924a1a982b30e5a85c581d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46781 } }
I20260928 05:21:53.171844 19184 leader_election.cc:302] T 00000000000000000000000000000000 P 34f88604b2924a1a982b30e5a85c581d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34f88604b2924a1a982b30e5a85c581d; no voters:
I20260928 05:21:53.172345 19184 leader_election.cc:288] T 00000000000000000000000000000000 P 34f88604b2924a1a982b30e5a85c581d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:21:53.172456 19189 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34f88604b2924a1a982b30e5a85c581d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:21:53.173082 19189 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34f88604b2924a1a982b30e5a85c581d [term 1 LEADER]: Becoming Leader. State: Replica: 34f88604b2924a1a982b30e5a85c581d, State: Running, Role: LEADER
I20260928 05:21:53.173542 19189 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34f88604b2924a1a982b30e5a85c581d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34f88604b2924a1a982b30e5a85c581d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46781 } }
I20260928 05:21:53.174209 19184 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34f88604b2924a1a982b30e5a85c581d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:21:53.175988 19190 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34f88604b2924a1a982b30e5a85c581d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "34f88604b2924a1a982b30e5a85c581d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34f88604b2924a1a982b30e5a85c581d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46781 } } }
I20260928 05:21:53.176370 19190 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34f88604b2924a1a982b30e5a85c581d [sys.catalog]: This master's current role is: LEADER
I20260928 05:21:53.177016 19191 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34f88604b2924a1a982b30e5a85c581d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34f88604b2924a1a982b30e5a85c581d. Latest consensus state: current_term: 1 leader_uuid: "34f88604b2924a1a982b30e5a85c581d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34f88604b2924a1a982b30e5a85c581d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46781 } } }
I20260928 05:21:53.177307 19191 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34f88604b2924a1a982b30e5a85c581d [sys.catalog]: This master's current role is: LEADER
I20260928 05:21:53.178121 19198 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:21:53.182934 19198 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:21:53.190765 19198 catalog_manager.cc:1347] Generated new cluster ID: 6deb2efd86d7400ebc1978b3960d1c65
I20260928 05:21:53.190871 19198 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:21:53.211815 19198 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:21:53.213229 19198 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:21:53.223450 19198 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 34f88604b2924a1a982b30e5a85c581d: Generated new TSK 0
I20260928 05:21:53.224185 19198 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 05:21:53.229650 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572622576160-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572622576160-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572622576160-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572622576160-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46781
--builtin_ntp_servers=127.0.97.20:41131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 05:21:53.541445 19208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:21:53.541698 19208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:21:53.541800 19208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:21:53.548781 19208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:21:53.548909 19208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:21:53.549058 19208 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 05:21:53.557919 19208 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572622576160-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572622576160-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572622576160-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46781
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572622576160-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:21:53.559120 19208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:21:53.560799 19208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:21:53.572822 19216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:21:53.572938 19213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:21:53.572952 19214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:21:53.574023 19208 server_base.cc:1034] running on GCE node
I20260928 05:21:53.574559 19208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:21:53.575456 19208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:21:53.576659 19208 hybrid_clock.cc:648] HybridClock initialized: now 1790572913576615 us; error 32 us; skew 500 ppm
I20260928 05:21:53.577026 19208 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 05:21:53.579155 19208 webserver.cc:460] Webserver started at http://127.0.97.1:37551/ using document root <none> and password file <none>
I20260928 05:21:53.579773 19208 fs_manager.cc:359] Metadata directory not provided
I20260928 05:21:53.579865 19208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:21:53.580160 19208 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:21:53.582842 19208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572622576160-388-0/minicluster-data/ts-0/data/instance:
uuid: "9c20316b34094d9fa080154b56153623"
format_stamp: "Formatted at 2026-09-28 05:21:53 on dist-test-slave-whsn"
server_key: "ed4b026b6344f8a530918c1a5e543362"
server_key_iv: "038318b2f7834442ff07701e07b80dec"
server_key_version: "encryptionkey@0"
I20260928 05:21:53.583573 19208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572622576160-388-0/minicluster-data/ts-0/wal/instance:
uuid: "9c20316b34094d9fa080154b56153623"
format_stamp: "Formatted at 2026-09-28 05:21:53 on dist-test-slave-whsn"
server_key: "ed4b026b6344f8a530918c1a5e543362"
server_key_iv: "038318b2f7834442ff07701e07b80dec"
server_key_version: "encryptionkey@0"
I20260928 05:21:53.588248 19208 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 05:21:53.591442 19222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:21:53.592630 19208 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260928 05:21:53.592768 19208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572622576160-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572622576160-388-0/minicluster-data/ts-0/wal
uuid: "9c20316b34094d9fa080154b56153623"
format_stamp: "Formatted at 2026-09-28 05:21:53 on dist-test-slave-whsn"
server_key: "ed4b026b6344f8a530918c1a5e543362"
server_key_iv: "038318b2f7834442ff07701e07b80dec"
server_key_version: "encryptionkey@0"
I20260928 05:21:53.593006 19208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572622576160-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572622576160-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572622576160-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:21:53.612670 19208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:21:53.613397 19208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:21:53.613633 19208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:21:53.614488 19208 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:21:53.615795 19208 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:21:53.615887 19208 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:21:53.615993 19208 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:21:53.616061 19208 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:21:53.645555 19208 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:46477
I20260928 05:21:53.645624 19334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:46477 every 8 connection(s)
I20260928 05:21:53.646763 19208 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572622576160-388-0/minicluster-data/ts-0/data/info.pb
I20260928 05:21:53.651336 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 19208
I20260928 05:21:53.651513 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572622576160-388-0/minicluster-data/ts-0/wal/instance
I20260928 05:21:53.652019 388 external_mini_cluster.cc:1428] Setting key c7612841496ed28f1abba630747e1948
I20260928 05:21:53.654717 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572622576160-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572622576160-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572622576160-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572622576160-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46781
--builtin_ntp_servers=127.0.97.20:41131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 05:21:53.659562 19187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:53.649063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52597 (local address 127.0.97.62:46781)
0928 05:21:53.649267 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 05:21:53.649273 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:21:53.650639 (+ 1366us) server_negotiation.cc:407] Connection header received
0928 05:21:53.652838 (+ 2199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:21:53.652843 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:21:53.652943 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:21:53.653057 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:21:53.654197 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:53.654923 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:53.657026 (+ 2103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:53.657422 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:53.657513 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:53.658881 (+ 1368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:21:53.658886 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:21:53.658891 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:21:53.658908 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:21:53.658943 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:21:53.658956 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:21:53.659033 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 05:21:53.659226 (+ 193us) server_negotiation.cc:299] Negotiation successful
0928 05:21:53.659312 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 05:21:53.660770 19337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:53.649619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46781 (local address 127.0.97.1:52597)
0928 05:21:53.650480 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:53.650518 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 05:21:53.652583 (+ 2065us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:21:53.653253 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:21:53.653261 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:21:53.653346 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:21:53.654031 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:53.654057 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:53.655069 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:53.655074 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:53.656883 (+ 1809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:53.656897 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:53.657619 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:53.657625 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:53.657878 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:53.658573 (+ 695us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 05:21:53.658576 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:21:53.658624 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 05:21:53.658711 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:21:53.659050 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:21:53.659058 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:21:53.659065 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 05:21:53.659216 (+ 151us) client_negotiation.cc:240] Negotiation successful
0928 05:21:53.659573 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":265,"threads_started":1}
I20260928 05:21:53.662123 19335 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46781
I20260928 05:21:53.662406 19335 heartbeater.cc:464] Registering TS with master...
I20260928 05:21:53.663225 19335 heartbeater.cc:511] Master 127.0.97.62:46781 requested a full tablet report, sending...
I20260928 05:21:53.665124 19149 ts_manager.cc:194] Registered new tserver with Master: 9c20316b34094d9fa080154b56153623 (127.0.97.1:46477)
I20260928 05:21:53.666561 19149 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:52597
W20260928 05:21:53.960651 19338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:21:53.960951 19338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:21:53.961087 19338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:21:53.968173 19338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:21:53.968266 19338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:21:53.968410 19338 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 05:21:53.977278 19338 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572622576160-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572622576160-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572622576160-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46781
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572622576160-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:21:53.978622 19338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:21:53.980348 19338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:21:53.992874 19343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:21:53.992905 19346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:21:53.992769 19344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:21:53.993834 19338 server_base.cc:1034] running on GCE node
I20260928 05:21:53.994398 19338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:21:53.995289 19338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:21:53.996480 19338 hybrid_clock.cc:648] HybridClock initialized: now 1790572913996436 us; error 40 us; skew 500 ppm
I20260928 05:21:53.996765 19338 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 05:21:53.998843 19338 webserver.cc:460] Webserver started at http://127.0.97.2:38415/ using document root <none> and password file <none>
I20260928 05:21:53.999470 19338 fs_manager.cc:359] Metadata directory not provided
I20260928 05:21:53.999549 19338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:21:53.999801 19338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:21:54.002512 19338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572622576160-388-0/minicluster-data/ts-1/data/instance:
uuid: "2b18bc5fc980420cac3462e00e01063b"
format_stamp: "Formatted at 2026-09-28 05:21:53 on dist-test-slave-whsn"
server_key: "4d78ec5e2aec7dc6af7068ca7ee55cae"
server_key_iv: "c4d33afc3d4e592535fdce0913434022"
server_key_version: "encryptionkey@0"
I20260928 05:21:54.003221 19338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572622576160-388-0/minicluster-data/ts-1/wal/instance:
uuid: "2b18bc5fc980420cac3462e00e01063b"
format_stamp: "Formatted at 2026-09-28 05:21:53 on dist-test-slave-whsn"
server_key: "4d78ec5e2aec7dc6af7068ca7ee55cae"
server_key_iv: "c4d33afc3d4e592535fdce0913434022"
server_key_version: "encryptionkey@0"
I20260928 05:21:54.007859 19338 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 05:21:54.010985 19352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:21:54.012193 19338 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 05:21:54.012328 19338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572622576160-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572622576160-388-0/minicluster-data/ts-1/wal
uuid: "2b18bc5fc980420cac3462e00e01063b"
format_stamp: "Formatted at 2026-09-28 05:21:53 on dist-test-slave-whsn"
server_key: "4d78ec5e2aec7dc6af7068ca7ee55cae"
server_key_iv: "c4d33afc3d4e592535fdce0913434022"
server_key_version: "encryptionkey@0"
I20260928 05:21:54.012508 19338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572622576160-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572622576160-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572622576160-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:21:54.025614 19338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:21:54.026290 19338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:21:54.026521 19338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:21:54.027359 19338 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:21:54.028677 19338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:21:54.028767 19338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:21:54.028887 19338 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:21:54.028959 19338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260928 05:21:54.057965 19338 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33321
I20260928 05:21:54.058027 19464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33321 every 8 connection(s)
I20260928 05:21:54.059172 19338 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572622576160-388-0/minicluster-data/ts-1/data/info.pb
I20260928 05:21:54.069249 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 19338
I20260928 05:21:54.069404 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572622576160-388-0/minicluster-data/ts-1/wal/instance
I20260928 05:21:54.069906 388 external_mini_cluster.cc:1428] Setting key 6752c67400c657ec855a42e054cf7684
I20260928 05:21:54.070533 19187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:54.061581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34705 (local address 127.0.97.62:46781)
0928 05:21:54.061798 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 05:21:54.061804 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:21:54.063045 (+ 1241us) server_negotiation.cc:407] Connection header received
0928 05:21:54.064686 (+ 1641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:21:54.064690 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:21:54.064755 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:21:54.064840 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:21:54.065851 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:54.066544 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:54.067978 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:54.068359 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:54.068448 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:54.069736 (+ 1288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:21:54.069743 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:21:54.069748 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:21:54.069770 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:21:54.069808 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:21:54.069825 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:21:54.069917 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 05:21:54.070148 (+ 231us) server_negotiation.cc:299] Negotiation successful
0928 05:21:54.070244 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 05:21:54.071440 19467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:54.062047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46781 (local address 127.0.97.2:34705)
0928 05:21:54.062883 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:54.062918 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 05:21:54.064465 (+ 1547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:21:54.065081 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:21:54.065089 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:21:54.065174 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:21:54.065715 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:54.065732 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:54.066662 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:54.066665 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:54.067860 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:54.067872 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:54.068522 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:54.068526 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:54.068717 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:54.069420 (+ 703us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 05:21:54.069424 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:21:54.069470 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 05:21:54.069559 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:21:54.069915 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:21:54.069921 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:21:54.069926 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 05:21:54.070102 (+ 176us) client_negotiation.cc:240] Negotiation successful
0928 05:21:54.070370 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":256,"threads_started":1}
I20260928 05:21:54.072654 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572622576160-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572622576160-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572622576160-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572622576160-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46781
--builtin_ntp_servers=127.0.97.20:41131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 05:21:54.072670 19465 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46781
I20260928 05:21:54.072962 19465 heartbeater.cc:464] Registering TS with master...
I20260928 05:21:54.073748 19465 heartbeater.cc:511] Master 127.0.97.62:46781 requested a full tablet report, sending...
I20260928 05:21:54.075162 19149 ts_manager.cc:194] Registered new tserver with Master: 2b18bc5fc980420cac3462e00e01063b (127.0.97.2:33321)
I20260928 05:21:54.076079 19149 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:34705
W20260928 05:21:54.377468 19468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:21:54.377774 19468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:21:54.377916 19468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:21:54.385298 19468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:21:54.385411 19468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:21:54.385545 19468 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 05:21:54.394330 19468 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572622576160-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572622576160-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572622576160-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46781
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572622576160-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:21:54.395540 19468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:21:54.397272 19468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:21:54.409145 19476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:21:54.409229 19474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:21:54.409322 19473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:21:54.410140 19468 server_base.cc:1034] running on GCE node
I20260928 05:21:54.410647 19468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:21:54.411484 19468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:21:54.412667 19468 hybrid_clock.cc:648] HybridClock initialized: now 1790572914412615 us; error 49 us; skew 500 ppm
I20260928 05:21:54.412971 19468 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 05:21:54.415007 19468 webserver.cc:460] Webserver started at http://127.0.97.3:36093/ using document root <none> and password file <none>
I20260928 05:21:54.415620 19468 fs_manager.cc:359] Metadata directory not provided
I20260928 05:21:54.415732 19468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:21:54.416021 19468 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:21:54.418751 19468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572622576160-388-0/minicluster-data/ts-2/data/instance:
uuid: "a56943f254bc4b759f226a626246e8f4"
format_stamp: "Formatted at 2026-09-28 05:21:54 on dist-test-slave-whsn"
server_key: "c070161f18bd5ec7cf3d6d179ab4d031"
server_key_iv: "046bb6e2250b618f43e9d4aab0708df1"
server_key_version: "encryptionkey@0"
I20260928 05:21:54.419469 19468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572622576160-388-0/minicluster-data/ts-2/wal/instance:
uuid: "a56943f254bc4b759f226a626246e8f4"
format_stamp: "Formatted at 2026-09-28 05:21:54 on dist-test-slave-whsn"
server_key: "c070161f18bd5ec7cf3d6d179ab4d031"
server_key_iv: "046bb6e2250b618f43e9d4aab0708df1"
server_key_version: "encryptionkey@0"
I20260928 05:21:54.423983 19468 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 05:21:54.427063 19482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:21:54.428232 19468 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 05:21:54.428368 19468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572622576160-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572622576160-388-0/minicluster-data/ts-2/wal
uuid: "a56943f254bc4b759f226a626246e8f4"
format_stamp: "Formatted at 2026-09-28 05:21:54 on dist-test-slave-whsn"
server_key: "c070161f18bd5ec7cf3d6d179ab4d031"
server_key_iv: "046bb6e2250b618f43e9d4aab0708df1"
server_key_version: "encryptionkey@0"
I20260928 05:21:54.428548 19468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572622576160-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572622576160-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572622576160-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:21:54.470338 19468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:21:54.471092 19468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:21:54.471328 19468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:21:54.472222 19468 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:21:54.473604 19468 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:21:54.473699 19468 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:21:54.473805 19468 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:21:54.473872 19468 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:21:54.504417 19468 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37587
I20260928 05:21:54.504455 19594 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37587 every 8 connection(s)
I20260928 05:21:54.505626 19468 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572622576160-388-0/minicluster-data/ts-2/data/info.pb
I20260928 05:21:54.506053 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 19468
I20260928 05:21:54.506219 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572622576160-388-0/minicluster-data/ts-2/wal/instance
I20260928 05:21:54.506752 388 external_mini_cluster.cc:1428] Setting key ea5a3c35329774ede517473db09efa1b
I20260928 05:21:54.518963 19187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:54.508024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51951 (local address 127.0.97.62:46781)
0928 05:21:54.508223 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 05:21:54.508229 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:21:54.509772 (+ 1543us) server_negotiation.cc:407] Connection header received
0928 05:21:54.511523 (+ 1751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:21:54.511527 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:21:54.511593 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:21:54.511669 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:21:54.512907 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:54.513927 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:54.516092 (+ 2165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:54.516693 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:54.516810 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:54.518169 (+ 1359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:21:54.518175 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:21:54.518180 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:21:54.518201 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:21:54.518238 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:21:54.518254 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:21:54.518336 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 05:21:54.518606 (+ 270us) server_negotiation.cc:299] Negotiation successful
0928 05:21:54.518689 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 05:21:54.520417 19597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:54.508619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46781 (local address 127.0.97.3:51951)
0928 05:21:54.509625 (+ 1006us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:54.509673 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 05:21:54.511313 (+ 1640us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:21:54.511847 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:21:54.511859 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:21:54.511945 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:21:54.512680 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:54.512704 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:54.514087 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:54.514093 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:54.515948 (+ 1855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:54.515962 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:54.516943 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:54.516951 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:54.517205 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:54.517851 (+ 646us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 05:21:54.517854 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:21:54.517907 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 05:21:54.518003 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:21:54.518361 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:21:54.518373 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:21:54.518381 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 05:21:54.519230 (+ 849us) client_negotiation.cc:240] Negotiation successful
0928 05:21:54.519533 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":324,"threads_started":1}
I20260928 05:21:54.521688 19595 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46781
I20260928 05:21:54.521920 19595 heartbeater.cc:464] Registering TS with master...
I20260928 05:21:54.522550 19595 heartbeater.cc:511] Master 127.0.97.62:46781 requested a full tablet report, sending...
I20260928 05:21:54.523715 19149 ts_manager.cc:194] Registered new tserver with Master: a56943f254bc4b759f226a626246e8f4 (127.0.97.3:37587)
I20260928 05:21:54.524453 19149 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:51951
I20260928 05:21:54.533661 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 05:21:54.543413 19187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:54.537670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40164 (local address 127.0.97.62:46781)
0928 05:21:54.537878 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 05:21:54.537885 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:21:54.538117 (+ 232us) server_negotiation.cc:407] Connection header received
0928 05:21:54.538281 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:21:54.538285 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:21:54.538359 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:21:54.538442 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:21:54.538935 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:54.539660 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:54.541065 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:54.541479 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:54.541587 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:54.542705 (+ 1118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:21:54.542709 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:21:54.542712 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:21:54.542730 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:21:54.542760 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:21:54.542771 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:21:54.542899 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 05:21:54.543083 (+ 184us) server_negotiation.cc:299] Negotiation successful
0928 05:21:54.543134 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 05:21:54.547042 19149 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40164:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 05:21:54.553725 19149 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 05:21:54.575156 19607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:54.565008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33321 (local address 127.0.0.1:54520)
0928 05:21:54.565957 (+ 949us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:54.566004 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 05:21:54.566155 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:21:54.567225 (+ 1070us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:21:54.567233 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:21:54.567263 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:21:54.567731 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:54.567743 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:54.569910 (+ 2167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:54.569915 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:54.572347 (+ 2432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:54.572362 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:54.574189 (+ 1827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:54.574195 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:54.574391 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:54.574447 (+ 56us) client_negotiation.cc:769] Sending connection context
0928 05:21:54.574541 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 05:21:54.574767 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":695,"mutex_wait_us":73,"thread_start_us":540,"threads_started":1}
I20260928 05:21:54.575629 19608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:54.565202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37587 (local address 127.0.0.1:52706)
0928 05:21:54.567329 (+ 2127us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:54.567346 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 05:21:54.567462 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:21:54.568321 (+ 859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:21:54.568325 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:21:54.568347 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:21:54.568738 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:54.568750 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:54.570559 (+ 1809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:54.570564 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:54.572904 (+ 2340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:54.572918 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:54.574933 (+ 2015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:54.574939 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:54.575137 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:54.575171 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 05:21:54.575248 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 05:21:54.575361 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2006,"thread_start_us":1864,"threads_started":1}
I20260928 05:21:54.576411 19610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:54.565832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52706 (local address 127.0.97.3:37587)
0928 05:21:54.566929 (+ 1097us) server_negotiation.cc:206] Beginning negotiation
0928 05:21:54.566936 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:21:54.567593 (+ 657us) server_negotiation.cc:407] Connection header received
0928 05:21:54.567695 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:21:54.567702 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:21:54.567911 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:21:54.568083 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:21:54.568950 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:54.570385 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:54.573105 (+ 2720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:54.574415 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:54.574510 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:54.574773 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0928 05:21:54.575910 (+ 1137us) server_negotiation.cc:299] Negotiation successful
0928 05:21:54.576069 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":942,"thread_start_us":1440,"threads_started":1}
I20260928 05:21:54.577116 19609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:54.565300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54520 (local address 127.0.97.2:33321)
0928 05:21:54.566536 (+ 1236us) server_negotiation.cc:206] Beginning negotiation
0928 05:21:54.566546 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 05:21:54.566575 (+ 29us) server_negotiation.cc:407] Connection header received
0928 05:21:54.566680 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:21:54.566687 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:21:54.566959 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:21:54.567918 (+ 959us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:21:54.568187 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:54.569761 (+ 1574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:54.572578 (+ 2817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:54.573965 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:54.576316 (+ 2351us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:54.576547 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0928 05:21:54.576618 (+ 71us) server_negotiation.cc:299] Negotiation successful
0928 05:21:54.576774 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1077,"thread_start_us":960,"threads_started":1}
I20260928 05:21:54.580920 19530 tablet_service.cc:1467] Processing CreateTablet for tablet 34a9d0f59717423099d69ed1240b627c (DEFAULT_TABLE table=test-table [id=e6e314c1b8784a918eb3322836df56ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:21:54.580806 19400 tablet_service.cc:1467] Processing CreateTablet for tablet 34a9d0f59717423099d69ed1240b627c (DEFAULT_TABLE table=test-table [id=e6e314c1b8784a918eb3322836df56ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:21:54.582136 19400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34a9d0f59717423099d69ed1240b627c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:21:54.582140 19530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34a9d0f59717423099d69ed1240b627c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:21:54.584271 19607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:54.567468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46477 (local address 127.0.0.1:52986)
0928 05:21:54.576048 (+ 8580us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:54.576076 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0928 05:21:54.576166 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:21:54.577940 (+ 1774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:21:54.577944 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:21:54.577970 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:21:54.578115 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:54.578126 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:54.580225 (+ 2099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:54.580230 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:54.581918 (+ 1688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:54.581932 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:54.583612 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:54.583618 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:54.583807 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:54.583840 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 05:21:54.583904 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 05:21:54.584025 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8492,"thread_start_us":9916,"threads_started":1}
I20260928 05:21:54.585305 19612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:54.567814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52986 (local address 127.0.97.1:46477)
0928 05:21:54.577295 (+ 9481us) server_negotiation.cc:206] Beginning negotiation
0928 05:21:54.577312 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 05:21:54.577342 (+ 30us) server_negotiation.cc:407] Connection header received
0928 05:21:54.577438 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:21:54.577445 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:21:54.577741 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:21:54.578313 (+ 572us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:21:54.578534 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:54.580063 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:54.582092 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:54.583423 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:54.584484 (+ 1061us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:54.584725 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0928 05:21:54.584790 (+ 65us) server_negotiation.cc:299] Negotiation successful
0928 05:21:54.584981 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9319,"thread_start_us":9197,"threads_started":1}
I20260928 05:21:54.588770 19270 tablet_service.cc:1467] Processing CreateTablet for tablet 34a9d0f59717423099d69ed1240b627c (DEFAULT_TABLE table=test-table [id=e6e314c1b8784a918eb3322836df56ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:21:54.590199 19270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34a9d0f59717423099d69ed1240b627c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:21:54.593538 19613 tablet_bootstrap.cc:492] T 34a9d0f59717423099d69ed1240b627c P a56943f254bc4b759f226a626246e8f4: Bootstrap starting.
I20260928 05:21:54.595203 19614 tablet_bootstrap.cc:492] T 34a9d0f59717423099d69ed1240b627c P 2b18bc5fc980420cac3462e00e01063b: Bootstrap starting.
I20260928 05:21:54.597226 19613 tablet_bootstrap.cc:654] T 34a9d0f59717423099d69ed1240b627c P a56943f254bc4b759f226a626246e8f4: Neither blocks nor log segments found. Creating new log.
I20260928 05:21:54.598532 19614 tablet_bootstrap.cc:654] T 34a9d0f59717423099d69ed1240b627c P 2b18bc5fc980420cac3462e00e01063b: Neither blocks nor log segments found. Creating new log.
I20260928 05:21:54.598524 19613 log.cc:824] T 34a9d0f59717423099d69ed1240b627c P a56943f254bc4b759f226a626246e8f4: Log is configured to *not* fsync() on all Append() calls
I20260928 05:21:54.599576 19614 log.cc:824] T 34a9d0f59717423099d69ed1240b627c P 2b18bc5fc980420cac3462e00e01063b: Log is configured to *not* fsync() on all Append() calls
I20260928 05:21:54.601341 19613 tablet_bootstrap.cc:492] T 34a9d0f59717423099d69ed1240b627c P a56943f254bc4b759f226a626246e8f4: No bootstrap required, opened a new log
I20260928 05:21:54.601610 19613 ts_tablet_manager.cc:1397] T 34a9d0f59717423099d69ed1240b627c P a56943f254bc4b759f226a626246e8f4: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260928 05:21:54.602346 19614 tablet_bootstrap.cc:492] T 34a9d0f59717423099d69ed1240b627c P 2b18bc5fc980420cac3462e00e01063b: No bootstrap required, opened a new log
I20260928 05:21:54.602650 19614 ts_tablet_manager.cc:1397] T 34a9d0f59717423099d69ed1240b627c P 2b18bc5fc980420cac3462e00e01063b: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.002s
I20260928 05:21:54.602790 19617 tablet_bootstrap.cc:492] T 34a9d0f59717423099d69ed1240b627c P 9c20316b34094d9fa080154b56153623: Bootstrap starting.
I20260928 05:21:54.606838 19617 tablet_bootstrap.cc:654] T 34a9d0f59717423099d69ed1240b627c P 9c20316b34094d9fa080154b56153623: Neither blocks nor log segments found. Creating new log.
I20260928 05:21:54.607671 19617 log.cc:824] T 34a9d0f59717423099d69ed1240b627c P 9c20316b34094d9fa080154b56153623: Log is configured to *not* fsync() on all Append() calls
I20260928 05:21:54.609876 19617 tablet_bootstrap.cc:492] T 34a9d0f59717423099d69ed1240b627c P 9c20316b34094d9fa080154b56153623: No bootstrap required, opened a new log
I20260928 05:21:54.610098 19617 ts_tablet_manager.cc:1397] T 34a9d0f59717423099d69ed1240b627c P 9c20316b34094d9fa080154b56153623: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260928 05:21:54.611776 19613 raft_consensus.cc:348] T 34a9d0f59717423099d69ed1240b627c P a56943f254bc4b759f226a626246e8f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b18bc5fc980420cac3462e00e01063b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33321 } } peers { permanent_uuid: "a56943f254bc4b759f226a626246e8f4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37587 } } peers { permanent_uuid: "9c20316b34094d9fa080154b56153623" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46477 } }
I20260928 05:21:54.612186 19613 raft_consensus.cc:374] T 34a9d0f59717423099d69ed1240b627c P a56943f254bc4b759f226a626246e8f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:21:54.612324 19613 raft_consensus.cc:729] T 34a9d0f59717423099d69ed1240b627c P a56943f254bc4b759f226a626246e8f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a56943f254bc4b759f226a626246e8f4, State: Initialized, Role: FOLLOWER
I20260928 05:21:54.612710 19614 raft_consensus.cc:348] T 34a9d0f59717423099d69ed1240b627c P 2b18bc5fc980420cac3462e00e01063b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b18bc5fc980420cac3462e00e01063b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33321 } } peers { permanent_uuid: "a56943f254bc4b759f226a626246e8f4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37587 } } peers { permanent_uuid: "9c20316b34094d9fa080154b56153623" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46477 } }
I20260928 05:21:54.612958 19613 consensus_queue.cc:260] T 34a9d0f59717423099d69ed1240b627c P a56943f254bc4b759f226a626246e8f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b18bc5fc980420cac3462e00e01063b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33321 } } peers { permanent_uuid: "a56943f254bc4b759f226a626246e8f4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37587 } } peers { permanent_uuid: "9c20316b34094d9fa080154b56153623" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46477 } }
I20260928 05:21:54.613183 19614 raft_consensus.cc:374] T 34a9d0f59717423099d69ed1240b627c P 2b18bc5fc980420cac3462e00e01063b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:21:54.613294 19614 raft_consensus.cc:729] T 34a9d0f59717423099d69ed1240b627c P 2b18bc5fc980420cac3462e00e01063b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b18bc5fc980420cac3462e00e01063b, State: Initialized, Role: FOLLOWER
I20260928 05:21:54.613925 19614 consensus_queue.cc:260] T 34a9d0f59717423099d69ed1240b627c P 2b18bc5fc980420cac3462e00e01063b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b18bc5fc980420cac3462e00e01063b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33321 } } peers { permanent_uuid: "a56943f254bc4b759f226a626246e8f4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37587 } } peers { permanent_uuid: "9c20316b34094d9fa080154b56153623" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46477 } }
I20260928 05:21:54.615002 19465 heartbeater.cc:503] Master 127.0.97.62:46781 was elected leader, sending a full tablet report...
I20260928 05:21:54.615757 19614 ts_tablet_manager.cc:1428] T 34a9d0f59717423099d69ed1240b627c P 2b18bc5fc980420cac3462e00e01063b: Time spent starting tablet: real 0.013s user 0.010s sys 0.002s
I20260928 05:21:54.617784 19595 heartbeater.cc:503] Master 127.0.97.62:46781 was elected leader, sending a full tablet report...
I20260928 05:21:54.618219 19613 ts_tablet_manager.cc:1428] T 34a9d0f59717423099d69ed1240b627c P a56943f254bc4b759f226a626246e8f4: Time spent starting tablet: real 0.016s user 0.008s sys 0.008s
I20260928 05:21:54.619344 19617 raft_consensus.cc:348] T 34a9d0f59717423099d69ed1240b627c P 9c20316b34094d9fa080154b56153623 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b18bc5fc980420cac3462e00e01063b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33321 } } peers { permanent_uuid: "a56943f254bc4b759f226a626246e8f4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37587 } } peers { permanent_uuid: "9c20316b34094d9fa080154b56153623" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46477 } }
I20260928 05:21:54.619637 19617 raft_consensus.cc:374] T 34a9d0f59717423099d69ed1240b627c P 9c20316b34094d9fa080154b56153623 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:21:54.619726 19617 raft_consensus.cc:729] T 34a9d0f59717423099d69ed1240b627c P 9c20316b34094d9fa080154b56153623 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c20316b34094d9fa080154b56153623, State: Initialized, Role: FOLLOWER
I20260928 05:21:54.620242 19617 consensus_queue.cc:260] T 34a9d0f59717423099d69ed1240b627c P 9c20316b34094d9fa080154b56153623 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b18bc5fc980420cac3462e00e01063b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33321 } } peers { permanent_uuid: "a56943f254bc4b759f226a626246e8f4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37587 } } peers { permanent_uuid: "9c20316b34094d9fa080154b56153623" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46477 } }
I20260928 05:21:54.621206 19335 heartbeater.cc:503] Master 127.0.97.62:46781 was elected leader, sending a full tablet report...
I20260928 05:21:54.621764 19617 ts_tablet_manager.cc:1428] T 34a9d0f59717423099d69ed1240b627c P 9c20316b34094d9fa080154b56153623: Time spent starting tablet: real 0.012s user 0.006s sys 0.005s
W20260928 05:21:54.648765 19336 tablet.cc:2369] T 34a9d0f59717423099d69ed1240b627c P 9c20316b34094d9fa080154b56153623: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 05:21:54.658061 19621 raft_consensus.cc:482] T 34a9d0f59717423099d69ed1240b627c P 9c20316b34094d9fa080154b56153623 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:21:54.658291 19621 raft_consensus.cc:504] T 34a9d0f59717423099d69ed1240b627c P 9c20316b34094d9fa080154b56153623 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b18bc5fc980420cac3462e00e01063b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33321 } } peers { permanent_uuid: "a56943f254bc4b759f226a626246e8f4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37587 } } peers { permanent_uuid: "9c20316b34094d9fa080154b56153623" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46477 } }
I20260928 05:21:54.659758 19621 leader_election.cc:288] T 34a9d0f59717423099d69ed1240b627c P 9c20316b34094d9fa080154b56153623 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2b18bc5fc980420cac3462e00e01063b (127.0.97.2:33321), a56943f254bc4b759f226a626246e8f4 (127.0.97.3:37587)
I20260928 05:21:54.661825 19620 raft_consensus.cc:482] T 34a9d0f59717423099d69ed1240b627c P 2b18bc5fc980420cac3462e00e01063b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:21:54.662096 19620 raft_consensus.cc:504] T 34a9d0f59717423099d69ed1240b627c P 2b18bc5fc980420cac3462e00e01063b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b18bc5fc980420cac3462e00e01063b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33321 } } peers { permanent_uuid: "a56943f254bc4b759f226a626246e8f4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37587 } } peers { permanent_uuid: "9c20316b34094d9fa080154b56153623" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46477 } }
I20260928 05:21:54.663801 19620 leader_election.cc:288] T 34a9d0f59717423099d69ed1240b627c P 2b18bc5fc980420cac3462e00e01063b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a56943f254bc4b759f226a626246e8f4 (127.0.97.3:37587), 9c20316b34094d9fa080154b56153623 (127.0.97.1:46477)
I20260928 05:21:54.668160 19609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:54.660198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54673 (local address 127.0.97.2:33321)
0928 05:21:54.660387 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 05:21:54.660394 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:21:54.661276 (+ 882us) server_negotiation.cc:407] Connection header received
0928 05:21:54.661412 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:21:54.661415 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:21:54.661473 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:21:54.661544 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:21:54.662214 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:54.663253 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:54.665713 (+ 2460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:54.666994 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:54.667092 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:54.667187 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 05:21:54.667750 (+ 563us) server_negotiation.cc:299] Negotiation successful
0928 05:21:54.667887 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 05:21:54.668823 19622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:54.660120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33321 (local address 127.0.97.1:54673)
0928 05:21:54.661181 (+ 1061us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:54.661196 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 05:21:54.661292 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:21:54.661595 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:21:54.661599 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:21:54.661622 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:21:54.662036 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:54.662050 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:54.663373 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:54.663382 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:54.665495 (+ 2113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:54.665511 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:54.667352 (+ 1841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:54.667359 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:54.667551 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:54.667586 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 05:21:54.668324 (+ 738us) client_negotiation.cc:240] Negotiation successful
0928 05:21:54.668471 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":957,"thread_start_us":436,"threads_started":1}
I20260928 05:21:54.669610 19610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:54.660213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58045 (local address 127.0.97.3:37587)
0928 05:21:54.660419 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 05:21:54.660426 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:21:54.660982 (+ 556us) server_negotiation.cc:407] Connection header received
0928 05:21:54.661193 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:21:54.661197 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:21:54.661254 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:21:54.661337 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:21:54.661924 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:54.662970 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:54.667141 (+ 4171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:54.668406 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:54.668503 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:54.668612 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 05:21:54.669210 (+ 598us) server_negotiation.cc:299] Negotiation successful
0928 05:21:54.669351 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 05:21:54.669955 19420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34a9d0f59717423099d69ed1240b627c" candidate_uuid: "9c20316b34094d9fa080154b56153623" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b18bc5fc980420cac3462e00e01063b" is_pre_election: true
I20260928 05:21:54.670450 19623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:54.660083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37587 (local address 127.0.97.1:58045)
0928 05:21:54.660876 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:54.660896 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 05:21:54.661051 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:21:54.661387 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:21:54.661392 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:21:54.661417 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:21:54.661783 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:54.661795 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:54.663118 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:54.663123 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:54.666924 (+ 3801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:54.666940 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:54.668788 (+ 1848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:54.668795 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:54.669007 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:54.669040 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 05:21:54.670007 (+ 967us) client_negotiation.cc:240] Negotiation successful
0928 05:21:54.670148 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":643,"thread_start_us":833,"threads_started":1}
I20260928 05:21:54.670401 19420 raft_consensus.cc:2454] T 34a9d0f59717423099d69ed1240b627c P 2b18bc5fc980420cac3462e00e01063b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c20316b34094d9fa080154b56153623 in term 0.
I20260928 05:21:54.671066 19223 leader_election.cc:302] T 34a9d0f59717423099d69ed1240b627c P 9c20316b34094d9fa080154b56153623 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b18bc5fc980420cac3462e00e01063b, 9c20316b34094d9fa080154b56153623; no voters:
I20260928 05:21:54.671563 19621 raft_consensus.cc:2781] T 34a9d0f59717423099d69ed1240b627c P 9c20316b34094d9fa080154b56153623 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 05:21:54.671701 19621 raft_consensus.cc:482] T 34a9d0f59717423099d69ed1240b627c P 9c20316b34094d9fa080154b56153623 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 05:21:54.671789 19621 raft_consensus.cc:3037] T 34a9d0f59717423099d69ed1240b627c P 9c20316b34094d9fa080154b56153623 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:21:54.671700 19550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34a9d0f59717423099d69ed1240b627c" candidate_uuid: "9c20316b34094d9fa080154b56153623" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a56943f254bc4b759f226a626246e8f4" is_pre_election: true
I20260928 05:21:54.672317 19550 raft_consensus.cc:2454] T 34a9d0f59717423099d69ed1240b627c P a56943f254bc4b759f226a626246e8f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c20316b34094d9fa080154b56153623 in term 0.
I20260928 05:21:54.674860 19621 raft_consensus.cc:504] T 34a9d0f59717423099d69ed1240b627c P 9c20316b34094d9fa080154b56153623 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b18bc5fc980420cac3462e00e01063b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33321 } } peers { permanent_uuid: "a56943f254bc4b759f226a626246e8f4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37587 } } peers { permanent_uuid: "9c20316b34094d9fa080154b56153623" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46477 } }
I20260928 05:21:54.675895 19550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34a9d0f59717423099d69ed1240b627c" candidate_uuid: "9c20316b34094d9fa080154b56153623" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a56943f254bc4b759f226a626246e8f4"
I20260928 05:21:54.675882 19420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34a9d0f59717423099d69ed1240b627c" candidate_uuid: "9c20316b34094d9fa080154b56153623" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b18bc5fc980420cac3462e00e01063b"
I20260928 05:21:54.676080 19420 raft_consensus.cc:3037] T 34a9d0f59717423099d69ed1240b627c P 2b18bc5fc980420cac3462e00e01063b [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:21:54.676095 19550 raft_consensus.cc:3037] T 34a9d0f59717423099d69ed1240b627c P a56943f254bc4b759f226a626246e8f4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:21:54.677217 19610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:54.664166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40695 (local address 127.0.97.3:37587)
0928 05:21:54.669823 (+ 5657us) server_negotiation.cc:206] Beginning negotiation
0928 05:21:54.669829 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:21:54.670157 (+ 328us) server_negotiation.cc:407] Connection header received
0928 05:21:54.670390 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:21:54.670395 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:21:54.670465 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:21:54.670536 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:21:54.671193 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:54.672256 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:54.674934 (+ 2678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:54.676106 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:54.676221 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:54.676317 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 05:21:54.676758 (+ 441us) server_negotiation.cc:299] Negotiation successful
0928 05:21:54.676925 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5557,"thread_start_us":6566,"threads_started":1}
I20260928 05:21:54.677953 19624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:54.663950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37587 (local address 127.0.97.2:40695)
0928 05:21:54.670051 (+ 6101us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:54.670076 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 05:21:54.670251 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:21:54.670694 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:21:54.670698 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:21:54.670721 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:21:54.671031 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:54.671043 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:54.672437 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:54.672442 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:54.674679 (+ 2237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:54.674694 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:54.676336 (+ 1642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:54.676342 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:54.676557 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:54.676604 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 05:21:54.677491 (+ 887us) client_negotiation.cc:240] Negotiation successful
0928 05:21:54.677642 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5961,"thread_start_us":7184,"threads_started":1}
I20260928 05:21:54.679049 19549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34a9d0f59717423099d69ed1240b627c" candidate_uuid: "2b18bc5fc980420cac3462e00e01063b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a56943f254bc4b759f226a626246e8f4" is_pre_election: true
I20260928 05:21:54.679316 19420 raft_consensus.cc:2454] T 34a9d0f59717423099d69ed1240b627c P 2b18bc5fc980420cac3462e00e01063b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c20316b34094d9fa080154b56153623 in term 1.
I20260928 05:21:54.679697 19223 leader_election.cc:302] T 34a9d0f59717423099d69ed1240b627c P 9c20316b34094d9fa080154b56153623 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b18bc5fc980420cac3462e00e01063b, 9c20316b34094d9fa080154b56153623; no voters:
I20260928 05:21:54.679716 19550 raft_consensus.cc:2454] T 34a9d0f59717423099d69ed1240b627c P a56943f254bc4b759f226a626246e8f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c20316b34094d9fa080154b56153623 in term 1.
I20260928 05:21:54.679880 19621 leader_election.cc:288] T 34a9d0f59717423099d69ed1240b627c P 9c20316b34094d9fa080154b56153623 [CANDIDATE]: Term 1 election: Requested vote from peers 2b18bc5fc980420cac3462e00e01063b (127.0.97.2:33321), a56943f254bc4b759f226a626246e8f4 (127.0.97.3:37587)
I20260928 05:21:54.680063 19621 raft_consensus.cc:2781] T 34a9d0f59717423099d69ed1240b627c P 9c20316b34094d9fa080154b56153623 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:21:54.680209 19621 raft_consensus.cc:686] T 34a9d0f59717423099d69ed1240b627c P 9c20316b34094d9fa080154b56153623 [term 1 LEADER]: Becoming Leader. State: Replica: 9c20316b34094d9fa080154b56153623, State: Running, Role: LEADER
I20260928 05:21:54.680768 19626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:54.671901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46477 (local address 127.0.97.2:38813)
0928 05:21:54.673745 (+ 1844us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:54.673759 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 05:21:54.673909 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:21:54.674221 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:21:54.674226 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:21:54.674249 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:21:54.674605 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:54.674613 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:54.676382 (+ 1769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:54.676386 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:54.678452 (+ 2066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:54.678464 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:54.680039 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:54.680045 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:54.680225 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:54.680259 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 05:21:54.680385 (+ 126us) client_negotiation.cc:240] Negotiation successful
0928 05:21:54.680494 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1743,"thread_start_us":1584,"threads_started":1}
I20260928 05:21:54.680905 19612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:54.671591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38813 (local address 127.0.97.1:46477)
0928 05:21:54.671761 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 05:21:54.671766 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:21:54.673865 (+ 2099us) server_negotiation.cc:407] Connection header received
0928 05:21:54.674032 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:21:54.674037 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:21:54.674111 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:21:54.674178 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:21:54.674782 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:54.675863 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:54.678640 (+ 2777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:54.679689 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:54.679823 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:54.679926 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 05:21:54.680420 (+ 494us) server_negotiation.cc:299] Negotiation successful
0928 05:21:54.680569 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 05:21:54.680727 19621 consensus_queue.cc:237] T 34a9d0f59717423099d69ed1240b627c P 9c20316b34094d9fa080154b56153623 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b18bc5fc980420cac3462e00e01063b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33321 } } peers { permanent_uuid: "a56943f254bc4b759f226a626246e8f4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37587 } } peers { permanent_uuid: "9c20316b34094d9fa080154b56153623" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46477 } }
I20260928 05:21:54.683529 19290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34a9d0f59717423099d69ed1240b627c" candidate_uuid: "2b18bc5fc980420cac3462e00e01063b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c20316b34094d9fa080154b56153623" is_pre_election: true
I20260928 05:21:54.684602 19356 leader_election.cc:302] T 34a9d0f59717423099d69ed1240b627c P 2b18bc5fc980420cac3462e00e01063b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2b18bc5fc980420cac3462e00e01063b; no voters: 9c20316b34094d9fa080154b56153623, a56943f254bc4b759f226a626246e8f4
I20260928 05:21:54.684960 19620 raft_consensus.cc:2726] T 34a9d0f59717423099d69ed1240b627c P 2b18bc5fc980420cac3462e00e01063b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 05:21:54.685196 19148 catalog_manager.cc:5495] T 34a9d0f59717423099d69ed1240b627c P 9c20316b34094d9fa080154b56153623 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c20316b34094d9fa080154b56153623 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "9c20316b34094d9fa080154b56153623" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b18bc5fc980420cac3462e00e01063b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a56943f254bc4b759f226a626246e8f4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c20316b34094d9fa080154b56153623" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46477 } health_report { overall_health: HEALTHY } } }
I20260928 05:21:54.700202 19612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:54.695479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52998 (local address 127.0.97.1:46477)
0928 05:21:54.695661 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 05:21:54.695666 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:21:54.695680 (+ 14us) server_negotiation.cc:407] Connection header received
0928 05:21:54.695830 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:21:54.695833 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:21:54.695907 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:21:54.696022 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:21:54.696312 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:54.697153 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:54.698414 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:54.698812 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:54.698894 (+ 82us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:54.699286 (+ 392us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:21:54.699737 (+ 451us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:21:54.699856 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 05:21:54.699967 (+ 111us) server_negotiation.cc:299] Negotiation successful
0928 05:21:54.699998 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 05:21:54.707415 19550 raft_consensus.cc:1261] T 34a9d0f59717423099d69ed1240b627c P a56943f254bc4b759f226a626246e8f4 [term 1 FOLLOWER]: Refusing update from remote peer 9c20316b34094d9fa080154b56153623: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:21:54.707459 19420 raft_consensus.cc:1261] T 34a9d0f59717423099d69ed1240b627c P 2b18bc5fc980420cac3462e00e01063b [term 1 FOLLOWER]: Refusing update from remote peer 9c20316b34094d9fa080154b56153623: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:21:54.708197 19627 consensus_queue.cc:1035] T 34a9d0f59717423099d69ed1240b627c P 9c20316b34094d9fa080154b56153623 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b18bc5fc980420cac3462e00e01063b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:21:54.708472 19621 consensus_queue.cc:1035] T 34a9d0f59717423099d69ed1240b627c P 9c20316b34094d9fa080154b56153623 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a56943f254bc4b759f226a626246e8f4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:21:54.731787 19149 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40164:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:21:54.732014 19149 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40164:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:21:54.735172 19149 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 34f88604b2924a1a982b30e5a85c581d: Sending DeleteTablet for 3 replicas of tablet 34a9d0f59717423099d69ed1240b627c
I20260928 05:21:54.736192 19400 tablet_service.cc:1514] Processing DeleteTablet for tablet 34a9d0f59717423099d69ed1240b627c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:21:54 UTC) from {username='slave'} at 127.0.0.1:54520
I20260928 05:21:54.736213 19530 tablet_service.cc:1514] Processing DeleteTablet for tablet 34a9d0f59717423099d69ed1240b627c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:21:54 UTC) from {username='slave'} at 127.0.0.1:52706
I20260928 05:21:54.736359 19270 tablet_service.cc:1514] Processing DeleteTablet for tablet 34a9d0f59717423099d69ed1240b627c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:21:54 UTC) from {username='slave'} at 127.0.0.1:52986
I20260928 05:21:54.738087 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 19208
I20260928 05:21:54.746295 19642 tablet_replica.cc:330] T 34a9d0f59717423099d69ed1240b627c P 9c20316b34094d9fa080154b56153623: stopping tablet replica
I20260928 05:21:54.746824 19642 raft_consensus.cc:2229] T 34a9d0f59717423099d69ed1240b627c P 9c20316b34094d9fa080154b56153623 [term 1 LEADER]: Raft consensus shutting down.
I20260928 05:21:54.747525 19642 raft_consensus.cc:2258] T 34a9d0f59717423099d69ed1240b627c P 9c20316b34094d9fa080154b56153623 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:21:54.749621 19642 ts_tablet_manager.cc:1905] T 34a9d0f59717423099d69ed1240b627c P 9c20316b34094d9fa080154b56153623: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:21:54.750157 19640 tablet_replica.cc:330] T 34a9d0f59717423099d69ed1240b627c P 2b18bc5fc980420cac3462e00e01063b: stopping tablet replica
I20260928 05:21:54.750676 19641 tablet_replica.cc:330] T 34a9d0f59717423099d69ed1240b627c P a56943f254bc4b759f226a626246e8f4: stopping tablet replica
I20260928 05:21:54.751048 19641 raft_consensus.cc:2229] T 34a9d0f59717423099d69ed1240b627c P a56943f254bc4b759f226a626246e8f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:21:54.751495 19641 raft_consensus.cc:2258] T 34a9d0f59717423099d69ed1240b627c P a56943f254bc4b759f226a626246e8f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:21:54.752107 19640 raft_consensus.cc:2229] T 34a9d0f59717423099d69ed1240b627c P 2b18bc5fc980420cac3462e00e01063b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:21:54.753157 19641 ts_tablet_manager.cc:1905] T 34a9d0f59717423099d69ed1240b627c P a56943f254bc4b759f226a626246e8f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:21:54.759305 19642 ts_tablet_manager.cc:1918] T 34a9d0f59717423099d69ed1240b627c P 9c20316b34094d9fa080154b56153623: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:21:54.759426 19641 ts_tablet_manager.cc:1918] T 34a9d0f59717423099d69ed1240b627c P a56943f254bc4b759f226a626246e8f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:21:54.759639 19641 log.cc:1196] T 34a9d0f59717423099d69ed1240b627c P a56943f254bc4b759f226a626246e8f4: Deleting WAL directory at /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572622576160-388-0/minicluster-data/ts-2/wal/wals/34a9d0f59717423099d69ed1240b627c
I20260928 05:21:54.759819 19640 raft_consensus.cc:2258] T 34a9d0f59717423099d69ed1240b627c P 2b18bc5fc980420cac3462e00e01063b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:21:54.760196 19641 ts_tablet_manager.cc:1939] T 34a9d0f59717423099d69ed1240b627c P a56943f254bc4b759f226a626246e8f4: Deleting consensus metadata
I20260928 05:21:54.760344 19642 log.cc:1196] T 34a9d0f59717423099d69ed1240b627c P 9c20316b34094d9fa080154b56153623: Deleting WAL directory at /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572622576160-388-0/minicluster-data/ts-0/wal/wals/34a9d0f59717423099d69ed1240b627c
I20260928 05:21:54.761186 19612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:54.739427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53014 (local address 127.0.97.1:46477)
0928 05:21:54.739611 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 05:21:54.739617 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:21:54.753325 (+ 13708us) server_negotiation.cc:407] Connection header received
0928 05:21:54.753605 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:21:54.753610 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:21:54.753708 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:21:54.753793 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:21:54.754363 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:54.755512 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:54.757349 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:54.758042 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:54.758132 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:54.759497 (+ 1365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:21:54.759507 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:21:54.759523 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:21:54.760311 (+ 788us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:21:54.760372 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:21:54.760404 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:21:54.760482 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 05:21:54.760745 (+ 263us) server_negotiation.cc:299] Negotiation successful
0928 05:21:54.760842 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 05:21:54.761762 19640 ts_tablet_manager.cc:1905] T 34a9d0f59717423099d69ed1240b627c P 2b18bc5fc980420cac3462e00e01063b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:21:54.762370 19330 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:21:54.762687 19136 catalog_manager.cc:4841] TS a56943f254bc4b759f226a626246e8f4 (127.0.97.3:37587): tablet 34a9d0f59717423099d69ed1240b627c (table test-table [id=e6e314c1b8784a918eb3322836df56ed]) successfully deleted
I20260928 05:21:54.767746 19640 ts_tablet_manager.cc:1918] T 34a9d0f59717423099d69ed1240b627c P 2b18bc5fc980420cac3462e00e01063b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:21:54.767877 19640 log.cc:1196] T 34a9d0f59717423099d69ed1240b627c P 2b18bc5fc980420cac3462e00e01063b: Deleting WAL directory at /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572622576160-388-0/minicluster-data/ts-1/wal/wals/34a9d0f59717423099d69ed1240b627c
I20260928 05:21:54.768257 19640 ts_tablet_manager.cc:1939] T 34a9d0f59717423099d69ed1240b627c P 2b18bc5fc980420cac3462e00e01063b: Deleting consensus metadata
I20260928 05:21:54.769663 19133 catalog_manager.cc:4841] TS 2b18bc5fc980420cac3462e00e01063b (127.0.97.2:33321): tablet 34a9d0f59717423099d69ed1240b627c (table test-table [id=e6e314c1b8784a918eb3322836df56ed]) successfully deleted
I20260928 05:21:54.855623 19642 ts_tablet_manager.cc:1939] T 34a9d0f59717423099d69ed1240b627c P 9c20316b34094d9fa080154b56153623: Deleting consensus metadata
I20260928 05:21:54.855870 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 19208
I20260928 05:21:54.870203 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 19338
W20260928 05:21:54.870108 19136 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:46477 (error 108)
I20260928 05:21:54.871833 19607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:54.871395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46477 (local address 127.0.0.1:53022)
0928 05:21:54.871598 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:54.871687 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260928 05:21:54.872042 19136 catalog_manager.cc:4584] TS 9c20316b34094d9fa080154b56153623 (127.0.97.1:46477): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 34a9d0f59717423099d69ed1240b627c: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46477: connect: Connection refused (error 111)
I20260928 05:21:54.876691 19609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:54.870980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54532 (local address 127.0.97.2:33321)
0928 05:21:54.871177 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 05:21:54.871182 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:21:54.871202 (+ 20us) server_negotiation.cc:407] Connection header received
0928 05:21:54.871266 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:21:54.871272 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:21:54.871362 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:21:54.871444 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:21:54.871853 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:54.872678 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:54.873929 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:54.874380 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:54.874489 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:54.875448 (+ 959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:21:54.875460 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:21:54.875476 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:21:54.876000 (+ 524us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:21:54.876053 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:21:54.876072 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:21:54.876175 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 05:21:54.876302 (+ 127us) server_negotiation.cc:299] Negotiation successful
0928 05:21:54.876398 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 05:21:54.877344 19460 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:21:54.925659 19607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:54.925215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46477 (local address 127.0.0.1:53034)
0928 05:21:54.925419 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:54.925506 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260928 05:21:54.926677 19607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:54.926301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46477 (local address 127.0.0.1:53048)
0928 05:21:54.926455 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:54.926538 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260928 05:21:54.965329 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 19338
I20260928 05:21:54.979132 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 19468
I20260928 05:21:54.985733 19610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:54.979861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52712 (local address 127.0.97.3:37587)
0928 05:21:54.980027 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 05:21:54.980033 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:21:54.980085 (+ 52us) server_negotiation.cc:407] Connection header received
0928 05:21:54.980287 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:21:54.980292 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:21:54.980380 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:21:54.980466 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:21:54.980916 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:54.981606 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:54.982893 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:54.983287 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:54.983394 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:54.984389 (+ 995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:21:54.984398 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:21:54.984413 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:21:54.985065 (+ 652us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:21:54.985114 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:21:54.985134 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:21:54.985230 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 05:21:54.985358 (+ 128us) server_negotiation.cc:299] Negotiation successful
0928 05:21:54.985448 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 05:21:54.986383 19590 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:21:54.986948 19607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:54.986433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46477 (local address 127.0.0.1:53062)
0928 05:21:54.986656 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:54.986780 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260928 05:21:54.987967 19607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:54.987572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46477 (local address 127.0.0.1:53074)
0928 05:21:54.987731 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:54.987810 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 05:21:55.068521 19607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:55.068052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46477 (local address 127.0.0.1:53086)
0928 05:21:55.068294 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:55.068380 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260928 05:21:55.069710 19607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:55.069284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46477 (local address 127.0.0.1:53096)
0928 05:21:55.069495 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:55.069570 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260928 05:21:55.073874 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 19468
I20260928 05:21:55.088069 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 19118
I20260928 05:21:55.089133 19179 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:21:55.195041 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 19118
2026-09-28T05:21:55Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2510 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572622576160-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:21:55 dist-test-slave-whsn krb5kdc[19651](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:21:55 dist-test-slave-whsn krb5kdc[19651](info): set up 2 sockets
Sep 28 05:21:55 dist-test-slave-whsn krb5kdc[19651](info): commencing operation
krb5kdc: starting...
W20260928 05:21:56.200680 388 mini_kdc.cc:121] Time spent starting KDC: real 0.970s user 0.005s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572622576160-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572622576160-388-0/krb5kdc/test-user.keytab.
Sep 28 05:21:56 dist-test-slave-whsn krb5kdc[19651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:21:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:21:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 05:21:56.423643 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572622576160-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572622576160-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572622576160-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34213
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34213
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:21:56.729821 19664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:21:56.730077 19664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:21:56.730202 19664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:21:56.737076 19664 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:21:56.737177 19664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:21:56.737246 19664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:21:56.737319 19664 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:21:56.737373 19664 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:21:56.746665 19664 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572622576160-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572622576160-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34213
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572622576160-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:21:56.747879 19664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:21:56.749183 19664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:21:56.759900 19670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:21:56.760027 19669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:21:56.760048 19672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:21:56.760767 19664 server_base.cc:1034] running on GCE node
I20260928 05:21:56.761554 19664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:21:56.762799 19664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:21:56.764029 19664 hybrid_clock.cc:648] HybridClock initialized: now 1790572916763946 us; error 46 us; skew 500 ppm
I20260928 05:21:56.764343 19664 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:21:56 dist-test-slave-whsn krb5kdc[19651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572916, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:21:56.770054 19664 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 05:21:56.771373 19664 webserver.cc:460] Webserver started at http://127.0.97.62:37497/ using document root <none> and password file <none>
I20260928 05:21:56.771987 19664 fs_manager.cc:359] Metadata directory not provided
I20260928 05:21:56.772091 19664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:21:56.772372 19664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:21:56.775206 19664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572622576160-388-0/minicluster-data/master-0/data/instance:
uuid: "58ad478ce6a646729ac71d9eb19f69c3"
format_stamp: "Formatted at 2026-09-28 05:21:56 on dist-test-slave-whsn"
server_key: "592046729f91b3b454b9c640a7df23dc"
server_key_iv: "37754cdb7275b3f7d610b144c91551b8"
server_key_version: "encryptionkey@0"
I20260928 05:21:56.775893 19664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572622576160-388-0/minicluster-data/master-0/wal/instance:
uuid: "58ad478ce6a646729ac71d9eb19f69c3"
format_stamp: "Formatted at 2026-09-28 05:21:56 on dist-test-slave-whsn"
server_key: "592046729f91b3b454b9c640a7df23dc"
server_key_iv: "37754cdb7275b3f7d610b144c91551b8"
server_key_version: "encryptionkey@0"
I20260928 05:21:56.780635 19664 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.006s
I20260928 05:21:56.783892 19679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:21:56.785292 19664 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:21:56.785444 19664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572622576160-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572622576160-388-0/minicluster-data/master-0/wal
uuid: "58ad478ce6a646729ac71d9eb19f69c3"
format_stamp: "Formatted at 2026-09-28 05:21:56 on dist-test-slave-whsn"
server_key: "592046729f91b3b454b9c640a7df23dc"
server_key_iv: "37754cdb7275b3f7d610b144c91551b8"
server_key_version: "encryptionkey@0"
I20260928 05:21:56.785642 19664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572622576160-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572622576160-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572622576160-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:21:56.808451 19664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:21:56.812537 19664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:21:56.812819 19664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:21:56.830948 19664 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34213
I20260928 05:21:56.830967 19730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34213 every 8 connection(s)
I20260928 05:21:56.832325 19664 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:21:56.832662 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 19664
I20260928 05:21:56.832828 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572622576160-388-0/minicluster-data/master-0/wal/instance
I20260928 05:21:56.833401 388 external_mini_cluster.cc:1428] Setting key 730a6c58b5bb999e7e93ec6a8df509f6
I20260928 05:21:56.836549 19731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 05:21:56 dist-test-slave-whsn krb5kdc[19651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:21:56.849062 19731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58ad478ce6a646729ac71d9eb19f69c3: Bootstrap starting.
I20260928 05:21:56.852188 19731 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 58ad478ce6a646729ac71d9eb19f69c3: Neither blocks nor log segments found. Creating new log.
I20260928 05:21:56.853389 19731 log.cc:824] T 00000000000000000000000000000000 P 58ad478ce6a646729ac71d9eb19f69c3: Log is configured to *not* fsync() on all Append() calls
I20260928 05:21:56.856242 19734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:56.835201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53156 (local address 127.0.97.62:34213)
0928 05:21:56.836011 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0928 05:21:56.836023 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 05:21:56.836065 (+ 42us) server_negotiation.cc:407] Connection header received
0928 05:21:56.837847 (+ 1782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:21:56.837870 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:21:56.838238 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:21:56.838565 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:21:56.840234 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:56.841721 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:56.843567 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:56.844266 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:56.844403 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:56.849071 (+ 4668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:21:56.849080 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:21:56.849094 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:21:56.849141 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:21:56.852128 (+ 2987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:21:56.852957 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:21:56.852964 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:21:56.852969 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:21:56.853157 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:21:56.853585 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:21:56.853591 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:21:56.853595 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:21:56.854255 (+ 660us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:21:56.854579 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0928 05:21:56.854900 (+ 321us) server_negotiation.cc:299] Negotiation successful
0928 05:21:56.855261 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":394,"threads_started":1}
I20260928 05:21:56.856350 19731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58ad478ce6a646729ac71d9eb19f69c3: No bootstrap required, opened a new log
I20260928 05:21:56.863353 19731 raft_consensus.cc:348] T 00000000000000000000000000000000 P 58ad478ce6a646729ac71d9eb19f69c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58ad478ce6a646729ac71d9eb19f69c3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34213 } }
I20260928 05:21:56.863687 19731 raft_consensus.cc:374] T 00000000000000000000000000000000 P 58ad478ce6a646729ac71d9eb19f69c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:21:56.863780 19731 raft_consensus.cc:729] T 00000000000000000000000000000000 P 58ad478ce6a646729ac71d9eb19f69c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58ad478ce6a646729ac71d9eb19f69c3, State: Initialized, Role: FOLLOWER
I20260928 05:21:56.864289 19731 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58ad478ce6a646729ac71d9eb19f69c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58ad478ce6a646729ac71d9eb19f69c3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34213 } }
I20260928 05:21:56.864485 19731 raft_consensus.cc:388] T 00000000000000000000000000000000 P 58ad478ce6a646729ac71d9eb19f69c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:21:56.864607 19731 raft_consensus.cc:482] T 00000000000000000000000000000000 P 58ad478ce6a646729ac71d9eb19f69c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:21:56.864777 19731 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 58ad478ce6a646729ac71d9eb19f69c3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:21:56.866825 19731 raft_consensus.cc:504] T 00000000000000000000000000000000 P 58ad478ce6a646729ac71d9eb19f69c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58ad478ce6a646729ac71d9eb19f69c3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34213 } }
I20260928 05:21:56.867249 19731 leader_election.cc:302] T 00000000000000000000000000000000 P 58ad478ce6a646729ac71d9eb19f69c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 58ad478ce6a646729ac71d9eb19f69c3; no voters:
I20260928 05:21:56.867753 19731 leader_election.cc:288] T 00000000000000000000000000000000 P 58ad478ce6a646729ac71d9eb19f69c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:21:56.867856 19736 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 58ad478ce6a646729ac71d9eb19f69c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:21:56.868451 19736 raft_consensus.cc:686] T 00000000000000000000000000000000 P 58ad478ce6a646729ac71d9eb19f69c3 [term 1 LEADER]: Becoming Leader. State: Replica: 58ad478ce6a646729ac71d9eb19f69c3, State: Running, Role: LEADER
I20260928 05:21:56.869014 19736 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58ad478ce6a646729ac71d9eb19f69c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58ad478ce6a646729ac71d9eb19f69c3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34213 } }
I20260928 05:21:56.869585 19731 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58ad478ce6a646729ac71d9eb19f69c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:21:56.871345 19737 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58ad478ce6a646729ac71d9eb19f69c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "58ad478ce6a646729ac71d9eb19f69c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58ad478ce6a646729ac71d9eb19f69c3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34213 } } }
I20260928 05:21:56.871706 19737 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58ad478ce6a646729ac71d9eb19f69c3 [sys.catalog]: This master's current role is: LEADER
I20260928 05:21:56.871908 19738 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58ad478ce6a646729ac71d9eb19f69c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58ad478ce6a646729ac71d9eb19f69c3. Latest consensus state: current_term: 1 leader_uuid: "58ad478ce6a646729ac71d9eb19f69c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58ad478ce6a646729ac71d9eb19f69c3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34213 } } }
I20260928 05:21:56.872202 19738 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58ad478ce6a646729ac71d9eb19f69c3 [sys.catalog]: This master's current role is: LEADER
I20260928 05:21:56.873790 19744 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:21:56.878296 19744 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:21:56.887423 19744 catalog_manager.cc:1347] Generated new cluster ID: 3439c3a8038b4cefa99754ebe86b5683
I20260928 05:21:56.887514 19744 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:21:56.912719 19744 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:21:56.914001 19744 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:21:56.920986 19744 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 58ad478ce6a646729ac71d9eb19f69c3: Generated new TSK 0
I20260928 05:21:56.921711 19744 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572622576160-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572622576160-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260928 05:21:56.984249 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572622576160-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572622576160-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572622576160-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572622576160-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34213
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572622576160-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:21:57.299226 19757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:21:57.299512 19757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:21:57.299666 19757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:21:57.306867 19757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:21:57.306970 19757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:21:57.307133 19757 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 05:21:57.317060 19757 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572622576160-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572622576160-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572622576160-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572622576160-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34213
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572622576160-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:21:57.318310 19757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:21:57.319602 19757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:21:57.332319 19763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:21:57.332362 19765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:21:57.332396 19762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:21:57.333079 19757 server_base.cc:1034] running on GCE node
I20260928 05:21:57.333654 19757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:21:57.334467 19757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:21:57.335664 19757 hybrid_clock.cc:648] HybridClock initialized: now 1790572917335628 us; error 25 us; skew 500 ppm
I20260928 05:21:57.335932 19757 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:21:57 dist-test-slave-whsn krb5kdc[19651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572917, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:21:57.338807 19757 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260928 05:21:57.340165 19757 webserver.cc:460] Webserver started at http://127.0.97.1:33155/ using document root <none> and password file <none>
I20260928 05:21:57.340742 19757 fs_manager.cc:359] Metadata directory not provided
I20260928 05:21:57.340834 19757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:21:57.341143 19757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:21:57.343813 19757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572622576160-388-0/minicluster-data/ts-0/data/instance:
uuid: "d57668c5710143fa915aff6cba598ee6"
format_stamp: "Formatted at 2026-09-28 05:21:57 on dist-test-slave-whsn"
server_key: "de3c76c06f1a4523db632a57a18832ee"
server_key_iv: "cba786b45fe563a09b06cdc30d08c21f"
server_key_version: "encryptionkey@0"
I20260928 05:21:57.344506 19757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572622576160-388-0/minicluster-data/ts-0/wal/instance:
uuid: "d57668c5710143fa915aff6cba598ee6"
format_stamp: "Formatted at 2026-09-28 05:21:57 on dist-test-slave-whsn"
server_key: "de3c76c06f1a4523db632a57a18832ee"
server_key_iv: "cba786b45fe563a09b06cdc30d08c21f"
server_key_version: "encryptionkey@0"
I20260928 05:21:57.349263 19757 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260928 05:21:57.352371 19772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:21:57.353626 19757 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 05:21:57.353765 19757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572622576160-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572622576160-388-0/minicluster-data/ts-0/wal
uuid: "d57668c5710143fa915aff6cba598ee6"
format_stamp: "Formatted at 2026-09-28 05:21:57 on dist-test-slave-whsn"
server_key: "de3c76c06f1a4523db632a57a18832ee"
server_key_iv: "cba786b45fe563a09b06cdc30d08c21f"
server_key_version: "encryptionkey@0"
I20260928 05:21:57.353942 19757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572622576160-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572622576160-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572622576160-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:21:57.385280 19757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:21:57.389075 19757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:21:57.389335 19757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:21:57.390280 19757 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:21:57.391534 19757 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:21:57.391605 19757 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:21:57.391723 19757 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:21:57.391772 19757 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:21:57.421705 19757 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45421
I20260928 05:21:57.421769 19884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45421 every 8 connection(s)
I20260928 05:21:57.422987 19757 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572622576160-388-0/minicluster-data/ts-0/data/info.pb
I20260928 05:21:57.425027 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 19757
I20260928 05:21:57.425155 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572622576160-388-0/minicluster-data/ts-0/wal/instance
I20260928 05:21:57.425578 388 external_mini_cluster.cc:1428] Setting key f4165cea45306f09f149007d8ba218c4
Sep 28 05:21:57 dist-test-slave-whsn krb5kdc[19651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572917, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:21:57.443519 19887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:57.425462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37937 (local address 127.0.97.62:34213)
0928 05:21:57.425919 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0928 05:21:57.425925 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:21:57.426961 (+ 1036us) server_negotiation.cc:407] Connection header received
0928 05:21:57.429269 (+ 2308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:21:57.429274 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:21:57.429344 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:21:57.429426 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:21:57.431224 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:57.432108 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:57.434174 (+ 2066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:57.434563 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:57.434663 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:57.438348 (+ 3685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:21:57.438353 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:21:57.438357 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:21:57.438386 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:21:57.440709 (+ 2323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:21:57.441499 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:21:57.441504 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:21:57.441506 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:21:57.441619 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:21:57.442029 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:21:57.442032 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:21:57.442034 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:21:57.442264 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:21:57.442366 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 05:21:57.443113 (+ 747us) server_negotiation.cc:299] Negotiation successful
0928 05:21:57.443280 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":226,"threads_started":1}
I20260928 05:21:57.444691 19888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:57.426018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34213 (local address 127.0.97.1:37937)
0928 05:21:57.426816 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:57.426851 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 05:21:57.428984 (+ 2133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:21:57.429691 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:21:57.429702 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:21:57.430350 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:21:57.431052 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:57.431075 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:57.432264 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:57.432268 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:57.434033 (+ 1765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:57.434047 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:57.434774 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:57.434779 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:57.435026 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:57.435655 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:21:57.435680 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:21:57.438152 (+ 2472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:21:57.440833 (+ 2681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:21:57.440840 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:21:57.440880 (+ 40us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:21:57.441374 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:21:57.441720 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:21:57.441724 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:21:57.441727 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:21:57.441932 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:21:57.442376 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:21:57.442383 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:21:57.442756 (+ 373us) client_negotiation.cc:769] Sending connection context
0928 05:21:57.443016 (+ 260us) client_negotiation.cc:240] Negotiation successful
0928 05:21:57.443485 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":324,"threads_started":1}
I20260928 05:21:57.446187 19885 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34213
I20260928 05:21:57.446456 19885 heartbeater.cc:464] Registering TS with master...
I20260928 05:21:57.447187 19885 heartbeater.cc:511] Master 127.0.97.62:34213 requested a full tablet report, sending...
I20260928 05:21:57.449245 19696 ts_manager.cc:194] Registered new tserver with Master: d57668c5710143fa915aff6cba598ee6 (127.0.97.1:45421)
I20260928 05:21:57.450628 19696 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:37937
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572622576160-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572622576160-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260928 05:21:57.479533 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572622576160-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572622576160-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572622576160-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572622576160-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34213
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572622576160-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:21:57.781832 19891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:21:57.782090 19891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:21:57.782182 19891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:21:57.789376 19891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:21:57.789551 19891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:21:57.789687 19891 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 05:21:57.799135 19891 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572622576160-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572622576160-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572622576160-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572622576160-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34213
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572622576160-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:21:57.800266 19891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:21:57.801456 19891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:21:57.813423 19897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:21:57.813465 19899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:21:57.813514 19896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:21:57.814256 19891 server_base.cc:1034] running on GCE node
I20260928 05:21:57.814778 19891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:21:57.815634 19891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:21:57.816875 19891 hybrid_clock.cc:648] HybridClock initialized: now 1790572917816810 us; error 23 us; skew 500 ppm
I20260928 05:21:57.817198 19891 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:21:57 dist-test-slave-whsn krb5kdc[19651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572917, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:21:57.820163 19891 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260928 05:21:57.821513 19891 webserver.cc:460] Webserver started at http://127.0.97.2:46425/ using document root <none> and password file <none>
I20260928 05:21:57.822131 19891 fs_manager.cc:359] Metadata directory not provided
I20260928 05:21:57.822242 19891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:21:57.822525 19891 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:21:57.825286 19891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572622576160-388-0/minicluster-data/ts-1/data/instance:
uuid: "9b0e0badfe15447d954cecbad17fc423"
format_stamp: "Formatted at 2026-09-28 05:21:57 on dist-test-slave-whsn"
server_key: "c2866895965e0e3ba01aa80dc22e4679"
server_key_iv: "858e06b2efe9f062bff973ec04eeac65"
server_key_version: "encryptionkey@0"
I20260928 05:21:57.825992 19891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572622576160-388-0/minicluster-data/ts-1/wal/instance:
uuid: "9b0e0badfe15447d954cecbad17fc423"
format_stamp: "Formatted at 2026-09-28 05:21:57 on dist-test-slave-whsn"
server_key: "c2866895965e0e3ba01aa80dc22e4679"
server_key_iv: "858e06b2efe9f062bff973ec04eeac65"
server_key_version: "encryptionkey@0"
I20260928 05:21:57.831066 19891 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 05:21:57.835238 19906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:21:57.836742 19891 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 05:21:57.836913 19891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572622576160-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572622576160-388-0/minicluster-data/ts-1/wal
uuid: "9b0e0badfe15447d954cecbad17fc423"
format_stamp: "Formatted at 2026-09-28 05:21:57 on dist-test-slave-whsn"
server_key: "c2866895965e0e3ba01aa80dc22e4679"
server_key_iv: "858e06b2efe9f062bff973ec04eeac65"
server_key_version: "encryptionkey@0"
I20260928 05:21:57.837126 19891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572622576160-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572622576160-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572622576160-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:21:57.869393 19891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:21:57.873174 19891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:21:57.873399 19891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:21:57.874277 19891 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:21:57.875459 19891 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:21:57.875527 19891 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:21:57.875618 19891 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:21:57.875659 19891 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:21:57.905253 19891 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33209
I20260928 05:21:57.905303 20018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33209 every 8 connection(s)
I20260928 05:21:57.906477 19891 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572622576160-388-0/minicluster-data/ts-1/data/info.pb
I20260928 05:21:57.909600 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 19891
I20260928 05:21:57.909751 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572622576160-388-0/minicluster-data/ts-1/wal/instance
I20260928 05:21:57.910169 388 external_mini_cluster.cc:1428] Setting key e8ac42bfbc7424118a308227e8046c53
Sep 28 05:21:57 dist-test-slave-whsn krb5kdc[19651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572917, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:21:57.926198 19887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:57.908977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58885 (local address 127.0.97.62:34213)
0928 05:21:57.909166 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 05:21:57.909171 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:21:57.910369 (+ 1198us) server_negotiation.cc:407] Connection header received
0928 05:21:57.912362 (+ 1993us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:21:57.912368 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:21:57.912450 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:21:57.912574 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:21:57.914106 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:57.915050 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:57.916544 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:57.917157 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:57.917273 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:57.920724 (+ 3451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:21:57.920729 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:21:57.920733 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:21:57.920764 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:21:57.923267 (+ 2503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:21:57.923948 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:21:57.923952 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:21:57.923954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:21:57.924079 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:21:57.924567 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:21:57.924571 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:21:57.924573 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:21:57.924827 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:21:57.924983 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 05:21:57.925649 (+ 666us) server_negotiation.cc:299] Negotiation successful
0928 05:21:57.925871 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 05:21:57.927161 20021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:57.909439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34213 (local address 127.0.97.2:58885)
0928 05:21:57.910207 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:57.910252 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 05:21:57.912104 (+ 1852us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:21:57.912805 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:21:57.912814 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:21:57.913372 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:21:57.913936 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:57.913953 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:57.915190 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:57.915194 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:57.916400 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:57.916412 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:57.917386 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:57.917391 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:57.917605 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:57.918254 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:21:57.918277 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:21:57.920510 (+ 2233us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:21:57.923411 (+ 2901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:21:57.923419 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:21:57.923435 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:21:57.923826 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:21:57.924197 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:21:57.924201 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:21:57.924205 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:21:57.924378 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:21:57.924963 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:21:57.924970 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:21:57.925304 (+ 334us) client_negotiation.cc:769] Sending connection context
0928 05:21:57.925540 (+ 236us) client_negotiation.cc:240] Negotiation successful
0928 05:21:57.926051 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":332,"threads_started":1}
I20260928 05:21:57.928644 20019 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34213
I20260928 05:21:57.928952 20019 heartbeater.cc:464] Registering TS with master...
I20260928 05:21:57.929644 20019 heartbeater.cc:511] Master 127.0.97.62:34213 requested a full tablet report, sending...
I20260928 05:21:57.931099 19696 ts_manager.cc:194] Registered new tserver with Master: 9b0e0badfe15447d954cecbad17fc423 (127.0.97.2:33209)
I20260928 05:21:57.932192 19696 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:58885
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572622576160-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572622576160-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260928 05:21:57.960649 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572622576160-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572622576160-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572622576160-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572622576160-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34213
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572622576160-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:21:58.265470 20024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:21:58.265733 20024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:21:58.265856 20024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:21:58.273046 20024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:21:58.273151 20024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:21:58.273298 20024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 05:21:58.282791 20024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572622576160-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572622576160-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572622576160-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572622576160-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34213
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572622576160-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:21:58.283978 20024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:21:58.285408 20024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:21:58.297804 20030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:21:58.297832 20032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:21:58.297935 20029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:21:58.298480 20024 server_base.cc:1034] running on GCE node
I20260928 05:21:58.298986 20024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:21:58.299798 20024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:21:58.300944 20024 hybrid_clock.cc:648] HybridClock initialized: now 1790572918300888 us; error 43 us; skew 500 ppm
I20260928 05:21:58.301232 20024 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:21:58 dist-test-slave-whsn krb5kdc[19651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572918, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:21:58.304071 20024 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260928 05:21:58.305405 20024 webserver.cc:460] Webserver started at http://127.0.97.3:35013/ using document root <none> and password file <none>
I20260928 05:21:58.306020 20024 fs_manager.cc:359] Metadata directory not provided
I20260928 05:21:58.306123 20024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:21:58.306396 20024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:21:58.309190 20024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572622576160-388-0/minicluster-data/ts-2/data/instance:
uuid: "3c8812d88c0942689a9a7161fb4c78f5"
format_stamp: "Formatted at 2026-09-28 05:21:58 on dist-test-slave-whsn"
server_key: "6ddf2511960483013c050ab469d134c5"
server_key_iv: "ebf7e872a480e01d3d0a9c1ddb622850"
server_key_version: "encryptionkey@0"
I20260928 05:21:58.309913 20024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572622576160-388-0/minicluster-data/ts-2/wal/instance:
uuid: "3c8812d88c0942689a9a7161fb4c78f5"
format_stamp: "Formatted at 2026-09-28 05:21:58 on dist-test-slave-whsn"
server_key: "6ddf2511960483013c050ab469d134c5"
server_key_iv: "ebf7e872a480e01d3d0a9c1ddb622850"
server_key_version: "encryptionkey@0"
I20260928 05:21:58.314472 20024 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 05:21:58.317714 20039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:21:58.318967 20024 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:21:58.319108 20024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572622576160-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572622576160-388-0/minicluster-data/ts-2/wal
uuid: "3c8812d88c0942689a9a7161fb4c78f5"
format_stamp: "Formatted at 2026-09-28 05:21:58 on dist-test-slave-whsn"
server_key: "6ddf2511960483013c050ab469d134c5"
server_key_iv: "ebf7e872a480e01d3d0a9c1ddb622850"
server_key_version: "encryptionkey@0"
I20260928 05:21:58.319283 20024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572622576160-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572622576160-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572622576160-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:21:58.335963 20024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:21:58.339613 20024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:21:58.339857 20024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:21:58.340758 20024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:21:58.342067 20024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:21:58.342139 20024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:21:58.342231 20024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:21:58.342278 20024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:21:58.371892 20024 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43003
I20260928 05:21:58.371948 20151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43003 every 8 connection(s)
I20260928 05:21:58.373131 20024 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572622576160-388-0/minicluster-data/ts-2/data/info.pb
I20260928 05:21:58.381494 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 20024
I20260928 05:21:58.381678 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572622576160-388-0/minicluster-data/ts-2/wal/instance
I20260928 05:21:58.382210 388 external_mini_cluster.cc:1428] Setting key 47f50f3bbc2ea92b162f209e43fb1eef
Sep 28 05:21:58 dist-test-slave-whsn krb5kdc[19651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572918, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:21:58.392527 19887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:58.375509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42383 (local address 127.0.97.62:34213)
0928 05:21:58.375682 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 05:21:58.375686 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:21:58.376820 (+ 1134us) server_negotiation.cc:407] Connection header received
0928 05:21:58.378682 (+ 1862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:21:58.378689 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:21:58.378781 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:21:58.378892 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:21:58.380276 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:58.381108 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:58.383125 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:58.383567 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:58.383700 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:58.387160 (+ 3460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:21:58.387165 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:21:58.387169 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:21:58.387205 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:21:58.389919 (+ 2714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:21:58.390548 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:21:58.390552 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:21:58.390554 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:21:58.390681 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:21:58.391059 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:21:58.391063 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:21:58.391065 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:21:58.391335 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:21:58.391449 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 05:21:58.392087 (+ 638us) server_negotiation.cc:299] Negotiation successful
0928 05:21:58.392256 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 05:21:58.393452 20154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:58.375956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34213 (local address 127.0.97.3:42383)
0928 05:21:58.376657 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:58.376695 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 05:21:58.378367 (+ 1672us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:21:58.379085 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:21:58.379093 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:21:58.379592 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:21:58.380111 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:58.380128 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:58.381288 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:58.381294 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:58.382973 (+ 1679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:58.382985 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:58.383791 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:58.383801 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:58.384028 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:58.384693 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:21:58.384721 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:21:58.386944 (+ 2223us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:21:58.390053 (+ 3109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:21:58.390060 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:21:58.390072 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:21:58.390454 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:21:58.390764 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:21:58.390767 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:21:58.390770 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:21:58.390931 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:21:58.391445 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:21:58.391451 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:21:58.391753 (+ 302us) client_negotiation.cc:769] Sending connection context
0928 05:21:58.391971 (+ 218us) client_negotiation.cc:240] Negotiation successful
0928 05:21:58.392351 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":278,"threads_started":1}
I20260928 05:21:58.394848 20152 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34213
I20260928 05:21:58.395131 20152 heartbeater.cc:464] Registering TS with master...
I20260928 05:21:58.395784 20152 heartbeater.cc:511] Master 127.0.97.62:34213 requested a full tablet report, sending...
I20260928 05:21:58.397279 19696 ts_manager.cc:194] Registered new tserver with Master: 3c8812d88c0942689a9a7161fb4c78f5 (127.0.97.3:43003)
I20260928 05:21:58.398090 19696 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:42383
I20260928 05:21:58.409198 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 05:21:58.425170 19887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:58.412991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53158 (local address 127.0.97.62:34213)
0928 05:21:58.413180 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 05:21:58.413187 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:21:58.413534 (+ 347us) server_negotiation.cc:407] Connection header received
0928 05:21:58.413649 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:21:58.413653 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:21:58.413712 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:21:58.413788 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:21:58.415056 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:58.415776 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:58.417065 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:58.417444 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:58.417539 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:58.419541 (+ 2002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:21:58.419546 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:21:58.419549 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:21:58.419578 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:21:58.422450 (+ 2872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:21:58.423051 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:21:58.423058 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:21:58.423062 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:21:58.423211 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:21:58.423548 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:21:58.423554 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:21:58.423557 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:21:58.424019 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:21:58.424275 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0928 05:21:58.424556 (+ 281us) server_negotiation.cc:299] Negotiation successful
0928 05:21:58.424783 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 05:21:58.428804 19696 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53158:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 05:21:58.435313 19696 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 05:21:58.453775 19885 heartbeater.cc:503] Master 127.0.97.62:34213 was elected leader, sending a full tablet report...
I20260928 05:21:58.469574 20165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:58.446447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45421 (local address 127.0.0.1:53134)
0928 05:21:58.457488 (+ 11041us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:58.458174 (+ 686us) client_negotiation.cc:174] Beginning negotiation
0928 05:21:58.458368 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:21:58.460824 (+ 2456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:21:58.460832 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:21:58.460886 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:21:58.461295 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:58.461307 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:58.463675 (+ 2368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:58.463681 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:58.466167 (+ 2486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:58.466185 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:58.468628 (+ 2443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:58.468635 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:58.468832 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:58.468916 (+ 84us) client_negotiation.cc:769] Sending connection context
0928 05:21:58.469035 (+ 119us) client_negotiation.cc:240] Negotiation successful
0928 05:21:58.469231 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10740,"thread_start_us":10575,"threads_started":1}
I20260928 05:21:58.470806 20166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:58.446577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53134 (local address 127.0.97.1:45421)
0928 05:21:58.459889 (+ 13312us) server_negotiation.cc:206] Beginning negotiation
0928 05:21:58.459899 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 05:21:58.459930 (+ 31us) server_negotiation.cc:407] Connection header received
0928 05:21:58.460033 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:21:58.460042 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:21:58.460322 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:21:58.460518 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:21:58.461702 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:58.463497 (+ 1795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:58.466704 (+ 3207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:58.468166 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:58.468291 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:58.468510 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0928 05:21:58.470324 (+ 1814us) server_negotiation.cc:299] Negotiation successful
0928 05:21:58.470497 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13120,"thread_start_us":14848,"threads_started":1}
I20260928 05:21:58.474179 20164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:58.446467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33209 (local address 127.0.0.1:34914)
0928 05:21:58.463946 (+ 17479us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:58.463967 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 05:21:58.464091 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:21:58.467466 (+ 3375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:21:58.467471 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:21:58.467494 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:21:58.467882 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:58.467895 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:58.469751 (+ 1856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:58.469756 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:58.471715 (+ 1959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:58.471729 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:58.473447 (+ 1718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:58.473453 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:58.473626 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:58.473649 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 05:21:58.473726 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 05:21:58.473844 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16070,"thread_start_us":18834,"threads_started":1}
I20260928 05:21:58.474262 20167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:58.446893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34914 (local address 127.0.97.2:33209)
0928 05:21:58.466671 (+ 19778us) server_negotiation.cc:206] Beginning negotiation
0928 05:21:58.466679 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:21:58.466705 (+ 26us) server_negotiation.cc:407] Connection header received
0928 05:21:58.466834 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:21:58.466842 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:21:58.467066 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:21:58.467254 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:21:58.468074 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:58.469597 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:58.471893 (+ 2296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:58.473250 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:58.473362 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:58.473552 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0928 05:21:58.473798 (+ 246us) server_negotiation.cc:299] Negotiation successful
0928 05:21:58.473956 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19635,"thread_start_us":19525,"threads_started":1}
I20260928 05:21:58.475983 19820 tablet_service.cc:1467] Processing CreateTablet for tablet 9d9043ab07dc409b8e29cc9f8a695c06 (DEFAULT_TABLE table=test-table [id=9b0c0e75ee8a479ab5755b3fa54f8136]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:21:58.477515 19820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d9043ab07dc409b8e29cc9f8a695c06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:21:58.477597 19954 tablet_service.cc:1467] Processing CreateTablet for tablet 9d9043ab07dc409b8e29cc9f8a695c06 (DEFAULT_TABLE table=test-table [id=9b0c0e75ee8a479ab5755b3fa54f8136]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:21:58.478570 20165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:58.459196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43003 (local address 127.0.0.1:47704)
0928 05:21:58.470045 (+ 10849us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:58.470075 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 05:21:58.470167 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:21:58.472233 (+ 2066us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:21:58.472237 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:21:58.472259 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:21:58.472395 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:58.472404 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:58.474151 (+ 1747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:58.474157 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:58.475971 (+ 1814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:58.475986 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:58.477721 (+ 1735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:58.477727 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:58.477954 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:58.477987 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 05:21:58.478082 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 05:21:58.478214 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10744,"thread_start_us":12757,"threads_started":1}
I20260928 05:21:58.478621 20169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:58.459515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47704 (local address 127.0.97.3:43003)
0928 05:21:58.471634 (+ 12119us) server_negotiation.cc:206] Beginning negotiation
0928 05:21:58.471642 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:21:58.471671 (+ 29us) server_negotiation.cc:407] Connection header received
0928 05:21:58.471762 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:21:58.471769 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:21:58.471998 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:21:58.472228 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:21:58.472562 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:58.473964 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:58.476186 (+ 2222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:58.477501 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:58.477686 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:58.477895 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0928 05:21:58.478149 (+ 254us) server_negotiation.cc:299] Negotiation successful
0928 05:21:58.478301 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11947,"thread_start_us":11826,"threads_started":1}
I20260928 05:21:58.478873 19954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d9043ab07dc409b8e29cc9f8a695c06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:21:58.484314 20087 tablet_service.cc:1467] Processing CreateTablet for tablet 9d9043ab07dc409b8e29cc9f8a695c06 (DEFAULT_TABLE table=test-table [id=9b0c0e75ee8a479ab5755b3fa54f8136]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:21:58.485723 20087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d9043ab07dc409b8e29cc9f8a695c06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:21:58.489395 20170 tablet_bootstrap.cc:492] T 9d9043ab07dc409b8e29cc9f8a695c06 P 9b0e0badfe15447d954cecbad17fc423: Bootstrap starting.
I20260928 05:21:58.492429 20170 tablet_bootstrap.cc:654] T 9d9043ab07dc409b8e29cc9f8a695c06 P 9b0e0badfe15447d954cecbad17fc423: Neither blocks nor log segments found. Creating new log.
I20260928 05:21:58.493724 20170 log.cc:824] T 9d9043ab07dc409b8e29cc9f8a695c06 P 9b0e0badfe15447d954cecbad17fc423: Log is configured to *not* fsync() on all Append() calls
I20260928 05:21:58.495133 20171 tablet_bootstrap.cc:492] T 9d9043ab07dc409b8e29cc9f8a695c06 P d57668c5710143fa915aff6cba598ee6: Bootstrap starting.
I20260928 05:21:58.497780 20170 tablet_bootstrap.cc:492] T 9d9043ab07dc409b8e29cc9f8a695c06 P 9b0e0badfe15447d954cecbad17fc423: No bootstrap required, opened a new log
I20260928 05:21:58.498108 20170 ts_tablet_manager.cc:1397] T 9d9043ab07dc409b8e29cc9f8a695c06 P 9b0e0badfe15447d954cecbad17fc423: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20260928 05:21:58.500568 20173 tablet_bootstrap.cc:492] T 9d9043ab07dc409b8e29cc9f8a695c06 P 3c8812d88c0942689a9a7161fb4c78f5: Bootstrap starting.
I20260928 05:21:58.501183 20171 tablet_bootstrap.cc:654] T 9d9043ab07dc409b8e29cc9f8a695c06 P d57668c5710143fa915aff6cba598ee6: Neither blocks nor log segments found. Creating new log.
I20260928 05:21:58.503041 20171 log.cc:824] T 9d9043ab07dc409b8e29cc9f8a695c06 P d57668c5710143fa915aff6cba598ee6: Log is configured to *not* fsync() on all Append() calls
I20260928 05:21:58.503790 20173 tablet_bootstrap.cc:654] T 9d9043ab07dc409b8e29cc9f8a695c06 P 3c8812d88c0942689a9a7161fb4c78f5: Neither blocks nor log segments found. Creating new log.
I20260928 05:21:58.504909 20173 log.cc:824] T 9d9043ab07dc409b8e29cc9f8a695c06 P 3c8812d88c0942689a9a7161fb4c78f5: Log is configured to *not* fsync() on all Append() calls
I20260928 05:21:58.506579 20171 tablet_bootstrap.cc:492] T 9d9043ab07dc409b8e29cc9f8a695c06 P d57668c5710143fa915aff6cba598ee6: No bootstrap required, opened a new log
I20260928 05:21:58.506829 20171 ts_tablet_manager.cc:1397] T 9d9043ab07dc409b8e29cc9f8a695c06 P d57668c5710143fa915aff6cba598ee6: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.000s
I20260928 05:21:58.507795 20173 tablet_bootstrap.cc:492] T 9d9043ab07dc409b8e29cc9f8a695c06 P 3c8812d88c0942689a9a7161fb4c78f5: No bootstrap required, opened a new log
I20260928 05:21:58.507679 20170 raft_consensus.cc:348] T 9d9043ab07dc409b8e29cc9f8a695c06 P 9b0e0badfe15447d954cecbad17fc423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d57668c5710143fa915aff6cba598ee6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45421 } } peers { permanent_uuid: "9b0e0badfe15447d954cecbad17fc423" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33209 } } peers { permanent_uuid: "3c8812d88c0942689a9a7161fb4c78f5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43003 } }
I20260928 05:21:58.508044 20173 ts_tablet_manager.cc:1397] T 9d9043ab07dc409b8e29cc9f8a695c06 P 3c8812d88c0942689a9a7161fb4c78f5: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.003s
I20260928 05:21:58.508100 20170 raft_consensus.cc:374] T 9d9043ab07dc409b8e29cc9f8a695c06 P 9b0e0badfe15447d954cecbad17fc423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:21:58.508185 20170 raft_consensus.cc:729] T 9d9043ab07dc409b8e29cc9f8a695c06 P 9b0e0badfe15447d954cecbad17fc423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b0e0badfe15447d954cecbad17fc423, State: Initialized, Role: FOLLOWER
I20260928 05:21:58.508796 20170 consensus_queue.cc:260] T 9d9043ab07dc409b8e29cc9f8a695c06 P 9b0e0badfe15447d954cecbad17fc423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d57668c5710143fa915aff6cba598ee6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45421 } } peers { permanent_uuid: "9b0e0badfe15447d954cecbad17fc423" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33209 } } peers { permanent_uuid: "3c8812d88c0942689a9a7161fb4c78f5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43003 } }
I20260928 05:21:58.509858 20019 heartbeater.cc:503] Master 127.0.97.62:34213 was elected leader, sending a full tablet report...
I20260928 05:21:58.510466 20170 ts_tablet_manager.cc:1428] T 9d9043ab07dc409b8e29cc9f8a695c06 P 9b0e0badfe15447d954cecbad17fc423: Time spent starting tablet: real 0.012s user 0.009s sys 0.002s
I20260928 05:21:58.517900 20173 raft_consensus.cc:348] T 9d9043ab07dc409b8e29cc9f8a695c06 P 3c8812d88c0942689a9a7161fb4c78f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d57668c5710143fa915aff6cba598ee6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45421 } } peers { permanent_uuid: "9b0e0badfe15447d954cecbad17fc423" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33209 } } peers { permanent_uuid: "3c8812d88c0942689a9a7161fb4c78f5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43003 } }
I20260928 05:21:58.517900 20171 raft_consensus.cc:348] T 9d9043ab07dc409b8e29cc9f8a695c06 P d57668c5710143fa915aff6cba598ee6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d57668c5710143fa915aff6cba598ee6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45421 } } peers { permanent_uuid: "9b0e0badfe15447d954cecbad17fc423" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33209 } } peers { permanent_uuid: "3c8812d88c0942689a9a7161fb4c78f5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43003 } }
I20260928 05:21:58.518332 20171 raft_consensus.cc:374] T 9d9043ab07dc409b8e29cc9f8a695c06 P d57668c5710143fa915aff6cba598ee6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:21:58.518332 20173 raft_consensus.cc:374] T 9d9043ab07dc409b8e29cc9f8a695c06 P 3c8812d88c0942689a9a7161fb4c78f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:21:58.518430 20173 raft_consensus.cc:729] T 9d9043ab07dc409b8e29cc9f8a695c06 P 3c8812d88c0942689a9a7161fb4c78f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c8812d88c0942689a9a7161fb4c78f5, State: Initialized, Role: FOLLOWER
I20260928 05:21:58.518429 20171 raft_consensus.cc:729] T 9d9043ab07dc409b8e29cc9f8a695c06 P d57668c5710143fa915aff6cba598ee6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d57668c5710143fa915aff6cba598ee6, State: Initialized, Role: FOLLOWER
I20260928 05:21:58.519030 20173 consensus_queue.cc:260] T 9d9043ab07dc409b8e29cc9f8a695c06 P 3c8812d88c0942689a9a7161fb4c78f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d57668c5710143fa915aff6cba598ee6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45421 } } peers { permanent_uuid: "9b0e0badfe15447d954cecbad17fc423" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33209 } } peers { permanent_uuid: "3c8812d88c0942689a9a7161fb4c78f5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43003 } }
I20260928 05:21:58.519021 20171 consensus_queue.cc:260] T 9d9043ab07dc409b8e29cc9f8a695c06 P d57668c5710143fa915aff6cba598ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d57668c5710143fa915aff6cba598ee6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45421 } } peers { permanent_uuid: "9b0e0badfe15447d954cecbad17fc423" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33209 } } peers { permanent_uuid: "3c8812d88c0942689a9a7161fb4c78f5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43003 } }
I20260928 05:21:58.520479 20152 heartbeater.cc:503] Master 127.0.97.62:34213 was elected leader, sending a full tablet report...
I20260928 05:21:58.520731 20171 ts_tablet_manager.cc:1428] T 9d9043ab07dc409b8e29cc9f8a695c06 P d57668c5710143fa915aff6cba598ee6: Time spent starting tablet: real 0.014s user 0.008s sys 0.005s
I20260928 05:21:58.521126 20173 ts_tablet_manager.cc:1428] T 9d9043ab07dc409b8e29cc9f8a695c06 P 3c8812d88c0942689a9a7161fb4c78f5: Time spent starting tablet: real 0.013s user 0.010s sys 0.005s
I20260928 05:21:58.550715 20178 raft_consensus.cc:482] T 9d9043ab07dc409b8e29cc9f8a695c06 P 3c8812d88c0942689a9a7161fb4c78f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:21:58.551015 20178 raft_consensus.cc:504] T 9d9043ab07dc409b8e29cc9f8a695c06 P 3c8812d88c0942689a9a7161fb4c78f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d57668c5710143fa915aff6cba598ee6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45421 } } peers { permanent_uuid: "9b0e0badfe15447d954cecbad17fc423" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33209 } } peers { permanent_uuid: "3c8812d88c0942689a9a7161fb4c78f5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43003 } }
I20260928 05:21:58.552702 20178 leader_election.cc:288] T 9d9043ab07dc409b8e29cc9f8a695c06 P 3c8812d88c0942689a9a7161fb4c78f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d57668c5710143fa915aff6cba598ee6 (127.0.97.1:45421), 9b0e0badfe15447d954cecbad17fc423 (127.0.97.2:33209)
I20260928 05:21:58.558029 20166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:58.553150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59525 (local address 127.0.97.1:45421)
0928 05:21:58.553329 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 05:21:58.553335 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:21:58.553354 (+ 19us) server_negotiation.cc:407] Connection header received
0928 05:21:58.553544 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:21:58.553549 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:21:58.553607 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:21:58.553677 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:21:58.554080 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:58.554781 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:58.556305 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:58.557130 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:58.557245 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:58.557339 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 05:21:58.557702 (+ 363us) server_negotiation.cc:299] Negotiation successful
0928 05:21:58.557801 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 05:21:58.558032 20154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:58.552915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45421 (local address 127.0.97.3:59525)
0928 05:21:58.553250 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:58.553270 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 05:21:58.553390 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:21:58.553733 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:21:58.553738 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:21:58.553767 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:21:58.553932 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:58.553940 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:58.554919 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:58.554923 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:58.556138 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:58.556152 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:58.557296 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:58.557302 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:58.557508 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:58.557542 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 05:21:58.557643 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 05:21:58.557769 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188}
I20260928 05:21:58.559420 20167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:58.553206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58469 (local address 127.0.97.2:33209)
0928 05:21:58.553420 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 05:21:58.553426 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:21:58.554101 (+ 675us) server_negotiation.cc:407] Connection header received
0928 05:21:58.554272 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:21:58.554276 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:21:58.554337 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:21:58.554422 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:21:58.555002 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:58.555694 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:58.557502 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:58.558647 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:58.558775 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:58.558856 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 05:21:58.559164 (+ 308us) server_negotiation.cc:299] Negotiation successful
0928 05:21:58.559252 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 05:21:58.559309 19840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d9043ab07dc409b8e29cc9f8a695c06" candidate_uuid: "3c8812d88c0942689a9a7161fb4c78f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d57668c5710143fa915aff6cba598ee6" is_pre_election: true
I20260928 05:21:58.559571 20179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:58.553235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33209 (local address 127.0.97.3:58469)
0928 05:21:58.553987 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0928 05:21:58.554003 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 05:21:58.554130 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:21:58.554450 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:21:58.554455 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:21:58.554490 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:21:58.554814 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:58.554824 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:58.555854 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:58.555858 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:58.557337 (+ 1479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:21:58.557350 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:21:58.558843 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:21:58.558849 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:21:58.559020 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:58.559052 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 05:21:58.559144 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 05:21:58.559255 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"thread_start_us":420,"threads_started":1}
I20260928 05:21:58.559900 19840 raft_consensus.cc:2454] T 9d9043ab07dc409b8e29cc9f8a695c06 P d57668c5710143fa915aff6cba598ee6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c8812d88c0942689a9a7161fb4c78f5 in term 0.
I20260928 05:21:58.560542 20042 leader_election.cc:302] T 9d9043ab07dc409b8e29cc9f8a695c06 P 3c8812d88c0942689a9a7161fb4c78f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c8812d88c0942689a9a7161fb4c78f5, d57668c5710143fa915aff6cba598ee6; no voters:
I20260928 05:21:58.560528 19974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d9043ab07dc409b8e29cc9f8a695c06" candidate_uuid: "3c8812d88c0942689a9a7161fb4c78f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b0e0badfe15447d954cecbad17fc423" is_pre_election: true
I20260928 05:21:58.560902 20178 raft_consensus.cc:2781] T 9d9043ab07dc409b8e29cc9f8a695c06 P 3c8812d88c0942689a9a7161fb4c78f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 05:21:58.561057 20178 raft_consensus.cc:482] T 9d9043ab07dc409b8e29cc9f8a695c06 P 3c8812d88c0942689a9a7161fb4c78f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 05:21:58.561002 19974 raft_consensus.cc:2454] T 9d9043ab07dc409b8e29cc9f8a695c06 P 9b0e0badfe15447d954cecbad17fc423 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c8812d88c0942689a9a7161fb4c78f5 in term 0.
I20260928 05:21:58.561161 20178 raft_consensus.cc:3037] T 9d9043ab07dc409b8e29cc9f8a695c06 P 3c8812d88c0942689a9a7161fb4c78f5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:21:58.563362 20178 raft_consensus.cc:504] T 9d9043ab07dc409b8e29cc9f8a695c06 P 3c8812d88c0942689a9a7161fb4c78f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d57668c5710143fa915aff6cba598ee6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45421 } } peers { permanent_uuid: "9b0e0badfe15447d954cecbad17fc423" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33209 } } peers { permanent_uuid: "3c8812d88c0942689a9a7161fb4c78f5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43003 } }
I20260928 05:21:58.563861 20178 leader_election.cc:288] T 9d9043ab07dc409b8e29cc9f8a695c06 P 3c8812d88c0942689a9a7161fb4c78f5 [CANDIDATE]: Term 1 election: Requested vote from peers d57668c5710143fa915aff6cba598ee6 (127.0.97.1:45421), 9b0e0badfe15447d954cecbad17fc423 (127.0.97.2:33209)
I20260928 05:21:58.564247 19840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d9043ab07dc409b8e29cc9f8a695c06" candidate_uuid: "3c8812d88c0942689a9a7161fb4c78f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d57668c5710143fa915aff6cba598ee6"
I20260928 05:21:58.564303 19974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d9043ab07dc409b8e29cc9f8a695c06" candidate_uuid: "3c8812d88c0942689a9a7161fb4c78f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b0e0badfe15447d954cecbad17fc423"
I20260928 05:21:58.564390 19840 raft_consensus.cc:3037] T 9d9043ab07dc409b8e29cc9f8a695c06 P d57668c5710143fa915aff6cba598ee6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:21:58.564476 19974 raft_consensus.cc:3037] T 9d9043ab07dc409b8e29cc9f8a695c06 P 9b0e0badfe15447d954cecbad17fc423 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:21:58.566443 19840 raft_consensus.cc:2454] T 9d9043ab07dc409b8e29cc9f8a695c06 P d57668c5710143fa915aff6cba598ee6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c8812d88c0942689a9a7161fb4c78f5 in term 1.
I20260928 05:21:58.566629 19974 raft_consensus.cc:2454] T 9d9043ab07dc409b8e29cc9f8a695c06 P 9b0e0badfe15447d954cecbad17fc423 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c8812d88c0942689a9a7161fb4c78f5 in term 1.
I20260928 05:21:58.566784 20042 leader_election.cc:302] T 9d9043ab07dc409b8e29cc9f8a695c06 P 3c8812d88c0942689a9a7161fb4c78f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c8812d88c0942689a9a7161fb4c78f5, d57668c5710143fa915aff6cba598ee6; no voters:
I20260928 05:21:58.567062 20178 raft_consensus.cc:2781] T 9d9043ab07dc409b8e29cc9f8a695c06 P 3c8812d88c0942689a9a7161fb4c78f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:21:58.567515 20178 raft_consensus.cc:686] T 9d9043ab07dc409b8e29cc9f8a695c06 P 3c8812d88c0942689a9a7161fb4c78f5 [term 1 LEADER]: Becoming Leader. State: Replica: 3c8812d88c0942689a9a7161fb4c78f5, State: Running, Role: LEADER
I20260928 05:21:58.568030 20178 consensus_queue.cc:237] T 9d9043ab07dc409b8e29cc9f8a695c06 P 3c8812d88c0942689a9a7161fb4c78f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d57668c5710143fa915aff6cba598ee6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45421 } } peers { permanent_uuid: "9b0e0badfe15447d954cecbad17fc423" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33209 } } peers { permanent_uuid: "3c8812d88c0942689a9a7161fb4c78f5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43003 } }
I20260928 05:21:58.572135 19696 catalog_manager.cc:5495] T 9d9043ab07dc409b8e29cc9f8a695c06 P 3c8812d88c0942689a9a7161fb4c78f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c8812d88c0942689a9a7161fb4c78f5 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "3c8812d88c0942689a9a7161fb4c78f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d57668c5710143fa915aff6cba598ee6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b0e0badfe15447d954cecbad17fc423" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c8812d88c0942689a9a7161fb4c78f5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43003 } health_report { overall_health: HEALTHY } } }
I20260928 05:21:58.608889 20169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:58.604203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47712 (local address 127.0.97.3:43003)
0928 05:21:58.604386 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 05:21:58.604391 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:21:58.604408 (+ 17us) server_negotiation.cc:407] Connection header received
0928 05:21:58.604549 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:21:58.604553 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:21:58.604630 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:21:58.604763 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:21:58.605077 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:58.605771 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:58.607062 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:58.607431 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:58.607515 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:58.607917 (+ 402us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:21:58.608398 (+ 481us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:21:58.608500 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 05:21:58.608565 (+ 65us) server_negotiation.cc:299] Negotiation successful
0928 05:21:58.608624 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 05:21:58.616559 19974 raft_consensus.cc:1261] T 9d9043ab07dc409b8e29cc9f8a695c06 P 9b0e0badfe15447d954cecbad17fc423 [term 1 FOLLOWER]: Refusing update from remote peer 3c8812d88c0942689a9a7161fb4c78f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:21:58.616556 19840 raft_consensus.cc:1261] T 9d9043ab07dc409b8e29cc9f8a695c06 P d57668c5710143fa915aff6cba598ee6 [term 1 FOLLOWER]: Refusing update from remote peer 3c8812d88c0942689a9a7161fb4c78f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:21:58.617450 20180 consensus_queue.cc:1035] T 9d9043ab07dc409b8e29cc9f8a695c06 P 3c8812d88c0942689a9a7161fb4c78f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d57668c5710143fa915aff6cba598ee6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:21:58.617831 20178 consensus_queue.cc:1035] T 9d9043ab07dc409b8e29cc9f8a695c06 P 3c8812d88c0942689a9a7161fb4c78f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b0e0badfe15447d954cecbad17fc423" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 05:21:58.624562 20153 tablet.cc:2369] T 9d9043ab07dc409b8e29cc9f8a695c06 P 3c8812d88c0942689a9a7161fb4c78f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 05:21:58.627931 20183 mvcc.cc:205] Tried to move back new op lower bound from 7334186674644320256 to 7334186674458431488. Current Snapshot: MvccSnapshot[applied={T|T < 7334186674644320256}]
I20260928 05:21:58.629174 20184 mvcc.cc:205] Tried to move back new op lower bound from 7334186674644320256 to 7334186674458431488. Current Snapshot: MvccSnapshot[applied={T|T < 7334186674644320256}]
I20260928 05:21:58.639930 19694 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53158:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:21:58.640239 19694 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53158:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:21:58.643967 19694 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 58ad478ce6a646729ac71d9eb19f69c3: Sending DeleteTablet for 3 replicas of tablet 9d9043ab07dc409b8e29cc9f8a695c06
I20260928 05:21:58.645033 19820 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d9043ab07dc409b8e29cc9f8a695c06 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:21:58 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53134
I20260928 05:21:58.645112 20087 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d9043ab07dc409b8e29cc9f8a695c06 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:21:58 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47704
I20260928 05:21:58.645714 19954 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d9043ab07dc409b8e29cc9f8a695c06 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:21:58 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34914
I20260928 05:21:58.649482 20193 tablet_replica.cc:330] T 9d9043ab07dc409b8e29cc9f8a695c06 P 3c8812d88c0942689a9a7161fb4c78f5: stopping tablet replica
I20260928 05:21:58.649940 20193 raft_consensus.cc:2229] T 9d9043ab07dc409b8e29cc9f8a695c06 P 3c8812d88c0942689a9a7161fb4c78f5 [term 1 LEADER]: Raft consensus shutting down.
I20260928 05:21:58.650620 20193 raft_consensus.cc:2258] T 9d9043ab07dc409b8e29cc9f8a695c06 P 3c8812d88c0942689a9a7161fb4c78f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:21:58.652093 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 19757
I20260928 05:21:58.652957 20193 ts_tablet_manager.cc:1905] T 9d9043ab07dc409b8e29cc9f8a695c06 P 3c8812d88c0942689a9a7161fb4c78f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:21:58.657483 20194 tablet_replica.cc:330] T 9d9043ab07dc409b8e29cc9f8a695c06 P d57668c5710143fa915aff6cba598ee6: stopping tablet replica
I20260928 05:21:58.657981 20194 raft_consensus.cc:2229] T 9d9043ab07dc409b8e29cc9f8a695c06 P d57668c5710143fa915aff6cba598ee6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:21:58.658601 20194 raft_consensus.cc:2258] T 9d9043ab07dc409b8e29cc9f8a695c06 P d57668c5710143fa915aff6cba598ee6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:21:58.660759 20194 ts_tablet_manager.cc:1905] T 9d9043ab07dc409b8e29cc9f8a695c06 P d57668c5710143fa915aff6cba598ee6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:21:58.661825 20195 tablet_replica.cc:330] T 9d9043ab07dc409b8e29cc9f8a695c06 P 9b0e0badfe15447d954cecbad17fc423: stopping tablet replica
I20260928 05:21:58.662243 20195 raft_consensus.cc:2229] T 9d9043ab07dc409b8e29cc9f8a695c06 P 9b0e0badfe15447d954cecbad17fc423 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:21:58.662736 20195 raft_consensus.cc:2258] T 9d9043ab07dc409b8e29cc9f8a695c06 P 9b0e0badfe15447d954cecbad17fc423 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:21:58.664803 20195 ts_tablet_manager.cc:1905] T 9d9043ab07dc409b8e29cc9f8a695c06 P 9b0e0badfe15447d954cecbad17fc423: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:21:58.670229 20193 ts_tablet_manager.cc:1918] T 9d9043ab07dc409b8e29cc9f8a695c06 P 3c8812d88c0942689a9a7161fb4c78f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:21:58.670459 20193 log.cc:1196] T 9d9043ab07dc409b8e29cc9f8a695c06 P 3c8812d88c0942689a9a7161fb4c78f5: Deleting WAL directory at /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572622576160-388-0/minicluster-data/ts-2/wal/wals/9d9043ab07dc409b8e29cc9f8a695c06
I20260928 05:21:58.671123 20193 ts_tablet_manager.cc:1939] T 9d9043ab07dc409b8e29cc9f8a695c06 P 3c8812d88c0942689a9a7161fb4c78f5: Deleting consensus metadata
I20260928 05:21:58.671270 20195 ts_tablet_manager.cc:1918] T 9d9043ab07dc409b8e29cc9f8a695c06 P 9b0e0badfe15447d954cecbad17fc423: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:21:58.671418 20195 log.cc:1196] T 9d9043ab07dc409b8e29cc9f8a695c06 P 9b0e0badfe15447d954cecbad17fc423: Deleting WAL directory at /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572622576160-388-0/minicluster-data/ts-1/wal/wals/9d9043ab07dc409b8e29cc9f8a695c06
I20260928 05:21:58.671908 20195 ts_tablet_manager.cc:1939] T 9d9043ab07dc409b8e29cc9f8a695c06 P 9b0e0badfe15447d954cecbad17fc423: Deleting consensus metadata
I20260928 05:21:58.673086 19682 catalog_manager.cc:4841] TS 3c8812d88c0942689a9a7161fb4c78f5 (127.0.97.3:43003): tablet 9d9043ab07dc409b8e29cc9f8a695c06 (table test-table [id=9b0c0e75ee8a479ab5755b3fa54f8136]) successfully deleted
I20260928 05:21:58.673626 19683 catalog_manager.cc:4841] TS 9b0e0badfe15447d954cecbad17fc423 (127.0.97.2:33209): tablet 9d9043ab07dc409b8e29cc9f8a695c06 (table test-table [id=9b0c0e75ee8a479ab5755b3fa54f8136]) successfully deleted
Sep 28 05:21:58 dist-test-slave-whsn krb5kdc[19651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260928 05:21:58.674870 20194 ts_tablet_manager.cc:1918] T 9d9043ab07dc409b8e29cc9f8a695c06 P d57668c5710143fa915aff6cba598ee6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:21:58.675107 20194 log.cc:1196] T 9d9043ab07dc409b8e29cc9f8a695c06 P d57668c5710143fa915aff6cba598ee6: Deleting WAL directory at /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572622576160-388-0/minicluster-data/ts-0/wal/wals/9d9043ab07dc409b8e29cc9f8a695c06
I20260928 05:21:58.675632 20194 ts_tablet_manager.cc:1939] T 9d9043ab07dc409b8e29cc9f8a695c06 P d57668c5710143fa915aff6cba598ee6: Deleting consensus metadata
I20260928 05:21:58.677345 19682 catalog_manager.cc:4841] TS d57668c5710143fa915aff6cba598ee6 (127.0.97.1:45421): tablet 9d9043ab07dc409b8e29cc9f8a695c06 (table test-table [id=9b0c0e75ee8a479ab5755b3fa54f8136]) successfully deleted
I20260928 05:21:58.681277 20166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:58.653131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53142 (local address 127.0.97.1:45421)
0928 05:21:58.653429 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0928 05:21:58.653435 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:21:58.664389 (+ 10954us) server_negotiation.cc:407] Connection header received
0928 05:21:58.664881 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:21:58.664887 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:21:58.664974 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:21:58.665061 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:21:58.666704 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:58.667899 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:58.669865 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:58.670543 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:58.670634 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:58.675576 (+ 4942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:21:58.675588 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:21:58.675600 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:21:58.675641 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:21:58.678708 (+ 3067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:21:58.679313 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:21:58.679320 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:21:58.679322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:21:58.679439 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:21:58.679800 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:21:58.679803 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:21:58.679806 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:21:58.680130 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:21:58.680415 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0928 05:21:58.680735 (+ 320us) server_negotiation.cc:299] Negotiation successful
0928 05:21:58.680956 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184}
I20260928 05:21:58.681907 19880 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:21:58.768952 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 19757
I20260928 05:21:58.783766 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 19891
Sep 28 05:21:58 dist-test-slave-whsn krb5kdc[19651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260928 05:21:58.798491 20167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:58.784488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34928 (local address 127.0.97.2:33209)
0928 05:21:58.784658 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 05:21:58.784663 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:21:58.784683 (+ 20us) server_negotiation.cc:407] Connection header received
0928 05:21:58.784790 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:21:58.784796 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:21:58.784903 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:21:58.784994 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:21:58.786149 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:58.786921 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:58.788198 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:58.788575 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:58.788680 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:58.792968 (+ 4288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:21:58.792986 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:21:58.793005 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:21:58.793041 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:21:58.795747 (+ 2706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:21:58.796550 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:21:58.796557 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:21:58.796559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:21:58.796674 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:21:58.797149 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:21:58.797152 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:21:58.797154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:21:58.797523 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:21:58.797751 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0928 05:21:58.798056 (+ 305us) server_negotiation.cc:299] Negotiation successful
0928 05:21:58.798217 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 05:21:58.799204 20014 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:21:58.899676 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 19891
I20260928 05:21:58.919479 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 20024
Sep 28 05:21:58 dist-test-slave-whsn krb5kdc[19651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260928 05:21:58.934680 20169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:58.920228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47722 (local address 127.0.97.3:43003)
0928 05:21:58.920400 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 05:21:58.920405 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:21:58.920432 (+ 27us) server_negotiation.cc:407] Connection header received
0928 05:21:58.920544 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:21:58.920549 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:21:58.920614 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:21:58.920705 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:21:58.921977 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:58.922824 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:58.924141 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:58.924608 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:58.924719 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:58.929007 (+ 4288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:21:58.929021 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:21:58.929036 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:21:58.929071 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:21:58.931926 (+ 2855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:21:58.932559 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:21:58.932565 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:21:58.932568 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:21:58.932724 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:21:58.933213 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:21:58.933216 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:21:58.933218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:21:58.933568 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:21:58.933875 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0928 05:21:58.934143 (+ 268us) server_negotiation.cc:299] Negotiation successful
0928 05:21:58.934347 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 05:21:58.935362 20147 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:21:59.055116 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 20024
I20260928 05:21:59.071123 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 19664
I20260928 05:21:59.072139 19726 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:21:59.181097 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 19664
2026-09-28T05:21:59Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (3987 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-28T05:21:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:21:59Z Disabled control of system clock
I20260928 05:21:59.247032 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572622576160-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572622576160-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572622576160-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34487
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:39669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34487
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 05:21:59.554045 20204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:21:59.554318 20204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:21:59.554419 20204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:21:59.561508 20204 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:21:59.561601 20204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:21:59.561645 20204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:21:59.561697 20204 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:21:59.561738 20204 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:21:59.570611 20204 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572622576160-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572622576160-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34487
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572622576160-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:21:59.571758 20204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:21:59.573496 20204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:21:59.584337 20209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:21:59.584472 20210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:21:59.584403 20212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:21:59.585215 20204 server_base.cc:1034] running on GCE node
I20260928 05:21:59.585917 20204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:21:59.587266 20204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:21:59.588429 20204 hybrid_clock.cc:648] HybridClock initialized: now 1790572919588381 us; error 34 us; skew 500 ppm
I20260928 05:21:59.588764 20204 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 05:21:59.590816 20204 webserver.cc:460] Webserver started at http://127.0.97.62:33879/ using document root <none> and password file <none>
I20260928 05:21:59.591446 20204 fs_manager.cc:359] Metadata directory not provided
I20260928 05:21:59.591534 20204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:21:59.591820 20204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:21:59.594555 20204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572622576160-388-0/minicluster-data/master-0/data/instance:
uuid: "d09294f1e77c410aa1fb302657f73df0"
format_stamp: "Formatted at 2026-09-28 05:21:59 on dist-test-slave-whsn"
server_key: "311b030bfc610e1dfb105f4e6f115bd0"
server_key_iv: "80cfecca4702846a36357019e569365c"
server_key_version: "encryptionkey@0"
I20260928 05:21:59.595291 20204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572622576160-388-0/minicluster-data/master-0/wal/instance:
uuid: "d09294f1e77c410aa1fb302657f73df0"
format_stamp: "Formatted at 2026-09-28 05:21:59 on dist-test-slave-whsn"
server_key: "311b030bfc610e1dfb105f4e6f115bd0"
server_key_iv: "80cfecca4702846a36357019e569365c"
server_key_version: "encryptionkey@0"
I20260928 05:21:59.599931 20204 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260928 05:21:59.603080 20218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:21:59.604300 20204 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260928 05:21:59.604444 20204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572622576160-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572622576160-388-0/minicluster-data/master-0/wal
uuid: "d09294f1e77c410aa1fb302657f73df0"
format_stamp: "Formatted at 2026-09-28 05:21:59 on dist-test-slave-whsn"
server_key: "311b030bfc610e1dfb105f4e6f115bd0"
server_key_iv: "80cfecca4702846a36357019e569365c"
server_key_version: "encryptionkey@0"
I20260928 05:21:59.604627 20204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572622576160-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572622576160-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572622576160-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:21:59.633499 20204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:21:59.634302 20204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:21:59.634565 20204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:21:59.650224 20204 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34487
I20260928 05:21:59.650249 20269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34487 every 8 connection(s)
I20260928 05:21:59.651423 20204 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:21:59.654980 20270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:21:59.656337 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 20204
I20260928 05:21:59.656497 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572622576160-388-0/minicluster-data/master-0/wal/instance
I20260928 05:21:59.657088 388 external_mini_cluster.cc:1428] Setting key 1b312921d64b2437d13a7564453b71fa
I20260928 05:21:59.667644 20270 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d09294f1e77c410aa1fb302657f73df0: Bootstrap starting.
I20260928 05:21:59.670027 20273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:21:59.658948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58056 (local address 127.0.97.62:34487)
0928 05:21:59.659923 (+ 975us) server_negotiation.cc:206] Beginning negotiation
0928 05:21:59.659937 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 05:21:59.659975 (+ 38us) server_negotiation.cc:407] Connection header received
0928 05:21:59.661600 (+ 1625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:21:59.661622 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:21:59.662093 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:21:59.662460 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:21:59.662972 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:59.664585 (+ 1613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:59.665958 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:21:59.666588 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:21:59.666771 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:21:59.667876 (+ 1105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:21:59.667883 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:21:59.667897 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:21:59.668411 (+ 514us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:21:59.668483 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:21:59.668503 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:21:59.668611 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 05:21:59.668823 (+ 212us) server_negotiation.cc:299] Negotiation successful
0928 05:21:59.669159 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":345,"threads_started":1}
I20260928 05:21:59.670836 20270 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d09294f1e77c410aa1fb302657f73df0: Neither blocks nor log segments found. Creating new log.
I20260928 05:21:59.671875 20270 log.cc:824] T 00000000000000000000000000000000 P d09294f1e77c410aa1fb302657f73df0: Log is configured to *not* fsync() on all Append() calls
I20260928 05:21:59.674383 20270 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d09294f1e77c410aa1fb302657f73df0: No bootstrap required, opened a new log
I20260928 05:21:59.681159 20270 raft_consensus.cc:348] T 00000000000000000000000000000000 P d09294f1e77c410aa1fb302657f73df0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d09294f1e77c410aa1fb302657f73df0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34487 } }
I20260928 05:21:59.681492 20270 raft_consensus.cc:374] T 00000000000000000000000000000000 P d09294f1e77c410aa1fb302657f73df0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:21:59.681586 20270 raft_consensus.cc:729] T 00000000000000000000000000000000 P d09294f1e77c410aa1fb302657f73df0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09294f1e77c410aa1fb302657f73df0, State: Initialized, Role: FOLLOWER
I20260928 05:21:59.682086 20270 consensus_queue.cc:260] T 00000000000000000000000000000000 P d09294f1e77c410aa1fb302657f73df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d09294f1e77c410aa1fb302657f73df0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34487 } }
I20260928 05:21:59.682302 20270 raft_consensus.cc:388] T 00000000000000000000000000000000 P d09294f1e77c410aa1fb302657f73df0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:21:59.682425 20270 raft_consensus.cc:482] T 00000000000000000000000000000000 P d09294f1e77c410aa1fb302657f73df0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:21:59.682579 20270 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d09294f1e77c410aa1fb302657f73df0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:21:59.684614 20270 raft_consensus.cc:504] T 00000000000000000000000000000000 P d09294f1e77c410aa1fb302657f73df0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d09294f1e77c410aa1fb302657f73df0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34487 } }
I20260928 05:21:59.685140 20270 leader_election.cc:302] T 00000000000000000000000000000000 P d09294f1e77c410aa1fb302657f73df0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d09294f1e77c410aa1fb302657f73df0; no voters:
I20260928 05:21:59.685673 20270 leader_election.cc:288] T 00000000000000000000000000000000 P d09294f1e77c410aa1fb302657f73df0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:21:59.685770 20275 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d09294f1e77c410aa1fb302657f73df0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:21:59.686410 20275 raft_consensus.cc:686] T 00000000000000000000000000000000 P d09294f1e77c410aa1fb302657f73df0 [term 1 LEADER]: Becoming Leader. State: Replica: d09294f1e77c410aa1fb302657f73df0, State: Running, Role: LEADER
I20260928 05:21:59.686875 20275 consensus_queue.cc:237] T 00000000000000000000000000000000 P d09294f1e77c410aa1fb302657f73df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d09294f1e77c410aa1fb302657f73df0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34487 } }
I20260928 05:21:59.687610 20270 sys_catalog.cc:564] T 00000000000000000000000000000000 P d09294f1e77c410aa1fb302657f73df0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:21:59.689602 20277 sys_catalog.cc:455] T 00000000000000000000000000000000 P d09294f1e77c410aa1fb302657f73df0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d09294f1e77c410aa1fb302657f73df0. Latest consensus state: current_term: 1 leader_uuid: "d09294f1e77c410aa1fb302657f73df0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d09294f1e77c410aa1fb302657f73df0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34487 } } }
I20260928 05:21:59.689639 20276 sys_catalog.cc:455] T 00000000000000000000000000000000 P d09294f1e77c410aa1fb302657f73df0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d09294f1e77c410aa1fb302657f73df0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d09294f1e77c410aa1fb302657f73df0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34487 } } }
I20260928 05:21:59.690048 20277 sys_catalog.cc:458] T 00000000000000000000000000000000 P d09294f1e77c410aa1fb302657f73df0 [sys.catalog]: This master's current role is: LEADER
I20260928 05:21:59.690121 20276 sys_catalog.cc:458] T 00000000000000000000000000000000 P d09294f1e77c410aa1fb302657f73df0 [sys.catalog]: This master's current role is: LEADER
I20260928 05:21:59.692381 20280 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:21:59.696806 20280 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:21:59.704934 20280 catalog_manager.cc:1347] Generated new cluster ID: 20ace55f318645398ab04830c266b5c2
I20260928 05:21:59.705047 20280 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:21:59.718739 20280 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:21:59.720379 20280 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:21:59.732758 20280 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d09294f1e77c410aa1fb302657f73df0: Generated new TSK 0
I20260928 05:21:59.733798 20280 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 05:21:59.738777 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572622576160-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572622576160-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572622576160-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572622576160-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34487
--builtin_ntp_servers=127.0.97.20:39669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 05:22:00.051460 20294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:22:00.051733 20294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:22:00.051842 20294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:22:00.058912 20294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:22:00.059005 20294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:22:00.059155 20294 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 05:22:00.067961 20294 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572622576160-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572622576160-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572622576160-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34487
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572622576160-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:22:00.069242 20294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:22:00.070935 20294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:22:00.085152 20299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:22:00.085216 20300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:22:00.085237 20302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:22:00.086551 20294 server_base.cc:1034] running on GCE node
I20260928 05:22:00.087137 20294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:22:00.088068 20294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:22:00.089326 20294 hybrid_clock.cc:648] HybridClock initialized: now 1790572920089248 us; error 55 us; skew 500 ppm
I20260928 05:22:00.089632 20294 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 05:22:00.091879 20294 webserver.cc:460] Webserver started at http://127.0.97.1:37079/ using document root <none> and password file <none>
I20260928 05:22:00.092618 20294 fs_manager.cc:359] Metadata directory not provided
I20260928 05:22:00.092715 20294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:22:00.093065 20294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:22:00.095966 20294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572622576160-388-0/minicluster-data/ts-0/data/instance:
uuid: "8eb12ff1288c41888d3e54ae0fdbcf78"
format_stamp: "Formatted at 2026-09-28 05:22:00 on dist-test-slave-whsn"
server_key: "722b7b2cb91ff104c29bce81b9ae0f9e"
server_key_iv: "f2628706d4f4099a279aa758288d3ebf"
server_key_version: "encryptionkey@0"
I20260928 05:22:00.096765 20294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572622576160-388-0/minicluster-data/ts-0/wal/instance:
uuid: "8eb12ff1288c41888d3e54ae0fdbcf78"
format_stamp: "Formatted at 2026-09-28 05:22:00 on dist-test-slave-whsn"
server_key: "722b7b2cb91ff104c29bce81b9ae0f9e"
server_key_iv: "f2628706d4f4099a279aa758288d3ebf"
server_key_version: "encryptionkey@0"
I20260928 05:22:00.101877 20294 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260928 05:22:00.105253 20308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:22:00.106494 20294 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 05:22:00.106639 20294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572622576160-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572622576160-388-0/minicluster-data/ts-0/wal
uuid: "8eb12ff1288c41888d3e54ae0fdbcf78"
format_stamp: "Formatted at 2026-09-28 05:22:00 on dist-test-slave-whsn"
server_key: "722b7b2cb91ff104c29bce81b9ae0f9e"
server_key_iv: "f2628706d4f4099a279aa758288d3ebf"
server_key_version: "encryptionkey@0"
I20260928 05:22:00.106861 20294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572622576160-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572622576160-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572622576160-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:22:00.127493 20294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:22:00.128322 20294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:22:00.128624 20294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:22:00.129633 20294 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:22:00.131191 20294 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:22:00.131289 20294 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:22:00.131397 20294 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:22:00.131489 20294 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:22:00.161705 20294 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41853
I20260928 05:22:00.161737 20420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41853 every 8 connection(s)
I20260928 05:22:00.162926 20294 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572622576160-388-0/minicluster-data/ts-0/data/info.pb
I20260928 05:22:00.171737 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 20294
I20260928 05:22:00.171890 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572622576160-388-0/minicluster-data/ts-0/wal/instance
I20260928 05:22:00.172384 388 external_mini_cluster.cc:1428] Setting key 580151069335db2ee8b1e4ab938425b4
I20260928 05:22:00.175235 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572622576160-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572622576160-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572622576160-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572622576160-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34487
--builtin_ntp_servers=127.0.97.20:39669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 05:22:00.175359 20273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:00.165576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57813 (local address 127.0.97.62:34487)
0928 05:22:00.165790 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:00.165797 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:00.167132 (+ 1335us) server_negotiation.cc:407] Connection header received
0928 05:22:00.168788 (+ 1656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:00.168793 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:00.168895 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:00.169005 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:00.169988 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:00.170718 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:00.172621 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:00.173072 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:00.173215 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:00.174624 (+ 1409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:00.174641 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:00.174645 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:22:00.174665 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:00.174703 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:22:00.174718 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:00.174805 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:00.174973 (+ 168us) server_negotiation.cc:299] Negotiation successful
0928 05:22:00.175068 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 05:22:00.176209 20423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:00.166071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34487 (local address 127.0.97.1:57813)
0928 05:22:00.166957 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:00.166991 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 05:22:00.168533 (+ 1542us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:22:00.169196 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:22:00.169204 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:22:00.169286 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:22:00.169823 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:00.169838 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:00.170853 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:00.170857 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:00.172492 (+ 1635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:00.172507 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:00.173281 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:00.173287 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:00.173547 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:00.174295 (+ 748us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 05:22:00.174298 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:22:00.174347 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 05:22:00.174431 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:22:00.174803 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:22:00.174809 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:22:00.174813 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 05:22:00.174935 (+ 122us) client_negotiation.cc:240] Negotiation successful
0928 05:22:00.175221 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":263,"threads_started":1}
I20260928 05:22:00.177500 20421 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34487
I20260928 05:22:00.177790 20421 heartbeater.cc:464] Registering TS with master...
I20260928 05:22:00.178553 20421 heartbeater.cc:511] Master 127.0.97.62:34487 requested a full tablet report, sending...
I20260928 05:22:00.180586 20235 ts_manager.cc:194] Registered new tserver with Master: 8eb12ff1288c41888d3e54ae0fdbcf78 (127.0.97.1:41853)
I20260928 05:22:00.182049 20235 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:57813
W20260928 05:22:00.485947 20424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:22:00.486223 20424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:22:00.486335 20424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:22:00.493479 20424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:22:00.493575 20424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:22:00.493742 20424 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 05:22:00.503573 20424 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572622576160-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572622576160-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572622576160-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34487
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572622576160-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:22:00.504796 20424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:22:00.506450 20424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:22:00.518292 20429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:22:00.518298 20430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:22:00.518399 20432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:22:00.518853 20424 server_base.cc:1034] running on GCE node
I20260928 05:22:00.519414 20424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:22:00.520216 20424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:22:00.521420 20424 hybrid_clock.cc:648] HybridClock initialized: now 1790572920521367 us; error 44 us; skew 500 ppm
I20260928 05:22:00.521692 20424 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 05:22:00.523669 20424 webserver.cc:460] Webserver started at http://127.0.97.2:34135/ using document root <none> and password file <none>
I20260928 05:22:00.524267 20424 fs_manager.cc:359] Metadata directory not provided
I20260928 05:22:00.524358 20424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:22:00.524641 20424 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:22:00.527372 20424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572622576160-388-0/minicluster-data/ts-1/data/instance:
uuid: "3ac6ba79e37b427796fefbb7b85f5ead"
format_stamp: "Formatted at 2026-09-28 05:22:00 on dist-test-slave-whsn"
server_key: "10c61563630ec0f5cf007562837aca14"
server_key_iv: "d04f7c464f04097b9b0be5a55c608006"
server_key_version: "encryptionkey@0"
I20260928 05:22:00.528062 20424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572622576160-388-0/minicluster-data/ts-1/wal/instance:
uuid: "3ac6ba79e37b427796fefbb7b85f5ead"
format_stamp: "Formatted at 2026-09-28 05:22:00 on dist-test-slave-whsn"
server_key: "10c61563630ec0f5cf007562837aca14"
server_key_iv: "d04f7c464f04097b9b0be5a55c608006"
server_key_version: "encryptionkey@0"
I20260928 05:22:00.532620 20424 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260928 05:22:00.535677 20438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:22:00.536818 20424 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 05:22:00.537015 20424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572622576160-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572622576160-388-0/minicluster-data/ts-1/wal
uuid: "3ac6ba79e37b427796fefbb7b85f5ead"
format_stamp: "Formatted at 2026-09-28 05:22:00 on dist-test-slave-whsn"
server_key: "10c61563630ec0f5cf007562837aca14"
server_key_iv: "d04f7c464f04097b9b0be5a55c608006"
server_key_version: "encryptionkey@0"
I20260928 05:22:00.537186 20424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572622576160-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572622576160-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572622576160-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:22:00.556679 20424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:22:00.557478 20424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:22:00.557700 20424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:22:00.558545 20424 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:22:00.559876 20424 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:22:00.559957 20424 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:22:00.560039 20424 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:22:00.560091 20424 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:22:00.589377 20424 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33711
I20260928 05:22:00.589440 20550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33711 every 8 connection(s)
I20260928 05:22:00.590631 20424 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572622576160-388-0/minicluster-data/ts-1/data/info.pb
I20260928 05:22:00.598034 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 20424
I20260928 05:22:00.598179 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572622576160-388-0/minicluster-data/ts-1/wal/instance
I20260928 05:22:00.598691 388 external_mini_cluster.cc:1428] Setting key 3aec3f494924eadfe52a5f48a950e03e
I20260928 05:22:00.601549 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572622576160-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572622576160-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572622576160-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572622576160-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34487
--builtin_ntp_servers=127.0.97.20:39669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 05:22:00.602864 20273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:00.592998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39215 (local address 127.0.97.62:34487)
0928 05:22:00.593165 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:00.593169 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:00.594531 (+ 1362us) server_negotiation.cc:407] Connection header received
0928 05:22:00.596203 (+ 1672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:00.596207 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:00.596280 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:00.596365 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:00.597470 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:00.598186 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:00.600077 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:00.600667 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:00.600763 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:00.602214 (+ 1451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:00.602219 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:00.602222 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:22:00.602238 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:00.602269 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:22:00.602280 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:00.602346 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:00.602577 (+ 231us) server_negotiation.cc:299] Negotiation successful
0928 05:22:00.602650 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 05:22:00.604072 20553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:00.593498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34487 (local address 127.0.97.2:39215)
0928 05:22:00.594350 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:00.594385 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 05:22:00.595957 (+ 1572us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:22:00.596539 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:22:00.596547 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:22:00.596628 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:22:00.597317 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:00.597335 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:00.598331 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:00.598336 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:00.599936 (+ 1600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:00.599948 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:00.600915 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:00.600923 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:00.601210 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:00.601872 (+ 662us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 05:22:00.601876 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:22:00.601920 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 05:22:00.602010 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:22:00.602368 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:22:00.602378 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:22:00.602385 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 05:22:00.602554 (+ 169us) client_negotiation.cc:240] Negotiation successful
0928 05:22:00.602913 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":237,"threads_started":1}
I20260928 05:22:00.605342 20551 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34487
I20260928 05:22:00.605623 20551 heartbeater.cc:464] Registering TS with master...
I20260928 05:22:00.606392 20551 heartbeater.cc:511] Master 127.0.97.62:34487 requested a full tablet report, sending...
I20260928 05:22:00.607728 20235 ts_manager.cc:194] Registered new tserver with Master: 3ac6ba79e37b427796fefbb7b85f5ead (127.0.97.2:33711)
I20260928 05:22:00.608528 20235 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:39215
W20260928 05:22:00.912693 20554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:22:00.913084 20554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:22:00.913225 20554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:22:00.924088 20554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:22:00.924196 20554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:22:00.924361 20554 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 05:22:00.937881 20554 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572622576160-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572622576160-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572622576160-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34487
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572622576160-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:22:00.939512 20554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:22:00.941854 20554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:22:00.956077 20560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:22:00.956053 20559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:22:00.956135 20562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:22:00.956717 20554 server_base.cc:1034] running on GCE node
I20260928 05:22:00.957283 20554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:22:00.958176 20554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:22:00.959389 20554 hybrid_clock.cc:648] HybridClock initialized: now 1790572920959348 us; error 37 us; skew 500 ppm
I20260928 05:22:00.959708 20554 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 05:22:00.961928 20554 webserver.cc:460] Webserver started at http://127.0.97.3:46109/ using document root <none> and password file <none>
I20260928 05:22:00.962579 20554 fs_manager.cc:359] Metadata directory not provided
I20260928 05:22:00.962664 20554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:22:00.962952 20554 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:22:00.965734 20554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572622576160-388-0/minicluster-data/ts-2/data/instance:
uuid: "1f27a46e93be4dacb46c0acd7adcc744"
format_stamp: "Formatted at 2026-09-28 05:22:00 on dist-test-slave-whsn"
server_key: "dc2f01ace4f2f6f3675bf1ec966bf992"
server_key_iv: "b7f0730b9c900bb1776f742e5e54b9d5"
server_key_version: "encryptionkey@0"
I20260928 05:22:00.966446 20554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572622576160-388-0/minicluster-data/ts-2/wal/instance:
uuid: "1f27a46e93be4dacb46c0acd7adcc744"
format_stamp: "Formatted at 2026-09-28 05:22:00 on dist-test-slave-whsn"
server_key: "dc2f01ace4f2f6f3675bf1ec966bf992"
server_key_iv: "b7f0730b9c900bb1776f742e5e54b9d5"
server_key_version: "encryptionkey@0"
I20260928 05:22:00.971798 20554 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 05:22:00.975275 20568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:22:00.976586 20554 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 05:22:00.976738 20554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572622576160-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572622576160-388-0/minicluster-data/ts-2/wal
uuid: "1f27a46e93be4dacb46c0acd7adcc744"
format_stamp: "Formatted at 2026-09-28 05:22:00 on dist-test-slave-whsn"
server_key: "dc2f01ace4f2f6f3675bf1ec966bf992"
server_key_iv: "b7f0730b9c900bb1776f742e5e54b9d5"
server_key_version: "encryptionkey@0"
I20260928 05:22:00.976979 20554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572622576160-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572622576160-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572622576160-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:22:00.994566 20554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:22:00.995290 20554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:22:00.995532 20554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:22:00.996415 20554 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:22:00.997828 20554 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:22:00.997920 20554 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:22:00.998034 20554 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:22:00.998101 20554 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:22:01.027484 20554 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43421
I20260928 05:22:01.027535 20680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43421 every 8 connection(s)
I20260928 05:22:01.028723 20554 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572622576160-388-0/minicluster-data/ts-2/data/info.pb
I20260928 05:22:01.034125 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 20554
I20260928 05:22:01.034253 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572622576160-388-0/minicluster-data/ts-2/wal/instance
I20260928 05:22:01.034729 388 external_mini_cluster.cc:1428] Setting key f6052b86ced8dcd94d71dbc6bc41d3b8
I20260928 05:22:01.040652 20273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:01.031170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56631 (local address 127.0.97.62:34487)
0928 05:22:01.031358 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:01.031363 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:01.032617 (+ 1254us) server_negotiation.cc:407] Connection header received
0928 05:22:01.034356 (+ 1739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:01.034362 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:01.034444 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:01.034550 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:01.035672 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:01.036535 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:01.038315 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:01.038715 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:01.038805 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:01.040033 (+ 1228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:01.040037 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:01.040041 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:22:01.040058 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:01.040096 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:22:01.040108 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:01.040179 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:01.040345 (+ 166us) server_negotiation.cc:299] Negotiation successful
0928 05:22:01.040415 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 05:22:01.041641 20683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:01.031603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34487 (local address 127.0.97.3:56631)
0928 05:22:01.032467 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:01.032508 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 05:22:01.034083 (+ 1575us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:22:01.034727 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:22:01.034735 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:22:01.034833 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:22:01.035502 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:01.035521 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:01.036681 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:01.036686 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:01.038182 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:01.038193 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:01.038926 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:01.038931 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:01.039136 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:01.039734 (+ 598us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 05:22:01.039736 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:22:01.039771 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 05:22:01.039846 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:22:01.040186 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:22:01.040193 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:22:01.040199 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 05:22:01.040328 (+ 129us) client_negotiation.cc:240] Negotiation successful
0928 05:22:01.040624 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":254,"threads_started":1}
I20260928 05:22:01.042893 20681 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34487
I20260928 05:22:01.043134 20681 heartbeater.cc:464] Registering TS with master...
I20260928 05:22:01.043782 20681 heartbeater.cc:511] Master 127.0.97.62:34487 requested a full tablet report, sending...
I20260928 05:22:01.045054 20235 ts_manager.cc:194] Registered new tserver with Master: 1f27a46e93be4dacb46c0acd7adcc744 (127.0.97.3:43421)
I20260928 05:22:01.045794 20235 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:56631
I20260928 05:22:01.049628 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 05:22:01.058635 20273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:01.053478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58072 (local address 127.0.97.62:34487)
0928 05:22:01.053685 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:01.053691 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:01.053969 (+ 278us) server_negotiation.cc:407] Connection header received
0928 05:22:01.054089 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:01.054093 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:01.054168 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:01.054245 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:01.054633 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:01.055346 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:01.056582 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:01.057021 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:01.057117 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:01.057998 (+ 881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:01.058003 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:01.058006 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:22:01.058023 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:01.058053 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:22:01.058064 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:01.058148 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:01.058250 (+ 102us) server_negotiation.cc:299] Negotiation successful
0928 05:22:01.058320 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 05:22:01.062120 20235 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58072:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 05:22:01.068688 20235 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 05:22:01.091180 20697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:01.082343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41853 (local address 127.0.0.1:37150)
0928 05:22:01.082781 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:01.082796 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 05:22:01.082893 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:22:01.083857 (+ 964us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:22:01.083862 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:22:01.083885 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:22:01.084250 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:01.084259 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:01.086024 (+ 1765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:01.086029 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:01.088719 (+ 2690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:01.088738 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:01.090494 (+ 1756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:01.090500 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:01.090656 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:01.090685 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 05:22:01.090766 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 05:22:01.090874 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"mutex_wait_us":340,"thread_start_us":606,"threads_started":1}
I20260928 05:22:01.091535 20693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:01.080464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33711 (local address 127.0.0.1:49078)
0928 05:22:01.082274 (+ 1810us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:01.082290 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 05:22:01.082388 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:22:01.083566 (+ 1178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:22:01.083575 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:22:01.083623 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:22:01.084069 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:01.084081 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:01.085907 (+ 1826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:01.085912 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:01.088777 (+ 2865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:01.088795 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:01.090107 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:01.090114 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:01.090328 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:01.090380 (+ 52us) client_negotiation.cc:769] Sending connection context
0928 05:22:01.090513 (+ 133us) client_negotiation.cc:240] Negotiation successful
0928 05:22:01.090683 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1694,"mutex_wait_us":293,"thread_start_us":702,"threads_started":1}
I20260928 05:22:01.091578 20698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:01.082795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37150 (local address 127.0.97.1:41853)
0928 05:22:01.083314 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:01.083321 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:01.083342 (+ 21us) server_negotiation.cc:407] Connection header received
0928 05:22:01.083413 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:01.083419 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:01.083661 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:01.084390 (+ 729us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:22:01.084548 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:01.085870 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:01.088940 (+ 3070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:01.090318 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:01.090440 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:01.090657 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:01.090825 (+ 168us) server_negotiation.cc:299] Negotiation successful
0928 05:22:01.090987 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":274,"threads_started":1}
I20260928 05:22:01.091915 20696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:01.080736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49078 (local address 127.0.97.2:33711)
0928 05:22:01.082386 (+ 1650us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:01.082395 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:01.082429 (+ 34us) server_negotiation.cc:407] Connection header received
0928 05:22:01.082532 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:01.082541 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:01.082825 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:01.084320 (+ 1495us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:22:01.084586 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:01.085765 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:01.089000 (+ 3235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:01.089904 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:01.090021 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:01.090299 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:01.091203 (+ 904us) server_negotiation.cc:299] Negotiation successful
0928 05:22:01.091457 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1497,"thread_start_us":1394,"threads_started":1}
I20260928 05:22:01.093575 20695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:01.079929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43421 (local address 127.0.0.1:58318)
0928 05:22:01.081595 (+ 1666us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:01.081676 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0928 05:22:01.081860 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:22:01.085859 (+ 3999us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:22:01.085864 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:22:01.085890 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:22:01.086370 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:01.086381 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:01.088049 (+ 1668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:01.088052 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:01.090122 (+ 2070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:01.090137 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:01.092816 (+ 2679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:01.092822 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:01.093011 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:01.093043 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 05:22:01.093135 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 05:22:01.093255 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1386,"mutex_wait_us":47,"thread_start_us":2064,"threads_started":1}
I20260928 05:22:01.093699 20694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:01.080175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58318 (local address 127.0.97.3:43421)
0928 05:22:01.085348 (+ 5173us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:01.085355 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:01.085386 (+ 31us) server_negotiation.cc:407] Connection header received
0928 05:22:01.085469 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:01.085476 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:01.085673 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:01.085832 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:22:01.086595 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:01.087915 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:01.090343 (+ 2428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:01.092643 (+ 2300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:01.092755 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:01.093036 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:01.093185 (+ 149us) server_negotiation.cc:299] Negotiation successful
0928 05:22:01.093397 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5003,"thread_start_us":4868,"threads_started":1}
I20260928 05:22:01.095427 20486 tablet_service.cc:1467] Processing CreateTablet for tablet 09c79774d5c64b85aa685f1d606846bc (DEFAULT_TABLE table=test-table [id=b9b78ed48914429087c5cd7fd56e1f18]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:22:01.095427 20356 tablet_service.cc:1467] Processing CreateTablet for tablet 09c79774d5c64b85aa685f1d606846bc (DEFAULT_TABLE table=test-table [id=b9b78ed48914429087c5cd7fd56e1f18]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:22:01.096840 20486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09c79774d5c64b85aa685f1d606846bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:22:01.096840 20356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09c79774d5c64b85aa685f1d606846bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:22:01.096892 20616 tablet_service.cc:1467] Processing CreateTablet for tablet 09c79774d5c64b85aa685f1d606846bc (DEFAULT_TABLE table=test-table [id=b9b78ed48914429087c5cd7fd56e1f18]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:22:01.097956 20616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09c79774d5c64b85aa685f1d606846bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:22:01.109436 20699 tablet_bootstrap.cc:492] T 09c79774d5c64b85aa685f1d606846bc P 1f27a46e93be4dacb46c0acd7adcc744: Bootstrap starting.
I20260928 05:22:01.110740 20700 tablet_bootstrap.cc:492] T 09c79774d5c64b85aa685f1d606846bc P 3ac6ba79e37b427796fefbb7b85f5ead: Bootstrap starting.
I20260928 05:22:01.112706 20699 tablet_bootstrap.cc:654] T 09c79774d5c64b85aa685f1d606846bc P 1f27a46e93be4dacb46c0acd7adcc744: Neither blocks nor log segments found. Creating new log.
I20260928 05:22:01.113970 20699 log.cc:824] T 09c79774d5c64b85aa685f1d606846bc P 1f27a46e93be4dacb46c0acd7adcc744: Log is configured to *not* fsync() on all Append() calls
I20260928 05:22:01.114022 20700 tablet_bootstrap.cc:654] T 09c79774d5c64b85aa685f1d606846bc P 3ac6ba79e37b427796fefbb7b85f5ead: Neither blocks nor log segments found. Creating new log.
I20260928 05:22:01.115320 20700 log.cc:824] T 09c79774d5c64b85aa685f1d606846bc P 3ac6ba79e37b427796fefbb7b85f5ead: Log is configured to *not* fsync() on all Append() calls
I20260928 05:22:01.116775 20701 tablet_bootstrap.cc:492] T 09c79774d5c64b85aa685f1d606846bc P 8eb12ff1288c41888d3e54ae0fdbcf78: Bootstrap starting.
I20260928 05:22:01.118257 20699 tablet_bootstrap.cc:492] T 09c79774d5c64b85aa685f1d606846bc P 1f27a46e93be4dacb46c0acd7adcc744: No bootstrap required, opened a new log
I20260928 05:22:01.118484 20699 ts_tablet_manager.cc:1397] T 09c79774d5c64b85aa685f1d606846bc P 1f27a46e93be4dacb46c0acd7adcc744: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260928 05:22:01.121145 20701 tablet_bootstrap.cc:654] T 09c79774d5c64b85aa685f1d606846bc P 8eb12ff1288c41888d3e54ae0fdbcf78: Neither blocks nor log segments found. Creating new log.
I20260928 05:22:01.122126 20701 log.cc:824] T 09c79774d5c64b85aa685f1d606846bc P 8eb12ff1288c41888d3e54ae0fdbcf78: Log is configured to *not* fsync() on all Append() calls
I20260928 05:22:01.123450 20700 tablet_bootstrap.cc:492] T 09c79774d5c64b85aa685f1d606846bc P 3ac6ba79e37b427796fefbb7b85f5ead: No bootstrap required, opened a new log
I20260928 05:22:01.123697 20700 ts_tablet_manager.cc:1397] T 09c79774d5c64b85aa685f1d606846bc P 3ac6ba79e37b427796fefbb7b85f5ead: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.006s
I20260928 05:22:01.124904 20701 tablet_bootstrap.cc:492] T 09c79774d5c64b85aa685f1d606846bc P 8eb12ff1288c41888d3e54ae0fdbcf78: No bootstrap required, opened a new log
I20260928 05:22:01.125145 20701 ts_tablet_manager.cc:1397] T 09c79774d5c64b85aa685f1d606846bc P 8eb12ff1288c41888d3e54ae0fdbcf78: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260928 05:22:01.126104 20699 raft_consensus.cc:348] T 09c79774d5c64b85aa685f1d606846bc P 1f27a46e93be4dacb46c0acd7adcc744 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f27a46e93be4dacb46c0acd7adcc744" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43421 } } peers { permanent_uuid: "8eb12ff1288c41888d3e54ae0fdbcf78" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41853 } } peers { permanent_uuid: "3ac6ba79e37b427796fefbb7b85f5ead" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33711 } }
I20260928 05:22:01.126425 20699 raft_consensus.cc:374] T 09c79774d5c64b85aa685f1d606846bc P 1f27a46e93be4dacb46c0acd7adcc744 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:22:01.126513 20699 raft_consensus.cc:729] T 09c79774d5c64b85aa685f1d606846bc P 1f27a46e93be4dacb46c0acd7adcc744 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f27a46e93be4dacb46c0acd7adcc744, State: Initialized, Role: FOLLOWER
I20260928 05:22:01.127065 20699 consensus_queue.cc:260] T 09c79774d5c64b85aa685f1d606846bc P 1f27a46e93be4dacb46c0acd7adcc744 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f27a46e93be4dacb46c0acd7adcc744" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43421 } } peers { permanent_uuid: "8eb12ff1288c41888d3e54ae0fdbcf78" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41853 } } peers { permanent_uuid: "3ac6ba79e37b427796fefbb7b85f5ead" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33711 } }
I20260928 05:22:01.129244 20681 heartbeater.cc:503] Master 127.0.97.62:34487 was elected leader, sending a full tablet report...
I20260928 05:22:01.129877 20699 ts_tablet_manager.cc:1428] T 09c79774d5c64b85aa685f1d606846bc P 1f27a46e93be4dacb46c0acd7adcc744: Time spent starting tablet: real 0.011s user 0.007s sys 0.005s
I20260928 05:22:01.133564 20700 raft_consensus.cc:348] T 09c79774d5c64b85aa685f1d606846bc P 3ac6ba79e37b427796fefbb7b85f5ead [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f27a46e93be4dacb46c0acd7adcc744" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43421 } } peers { permanent_uuid: "8eb12ff1288c41888d3e54ae0fdbcf78" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41853 } } peers { permanent_uuid: "3ac6ba79e37b427796fefbb7b85f5ead" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33711 } }
I20260928 05:22:01.133970 20700 raft_consensus.cc:374] T 09c79774d5c64b85aa685f1d606846bc P 3ac6ba79e37b427796fefbb7b85f5ead [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:22:01.134083 20700 raft_consensus.cc:729] T 09c79774d5c64b85aa685f1d606846bc P 3ac6ba79e37b427796fefbb7b85f5ead [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ac6ba79e37b427796fefbb7b85f5ead, State: Initialized, Role: FOLLOWER
I20260928 05:22:01.134642 20700 consensus_queue.cc:260] T 09c79774d5c64b85aa685f1d606846bc P 3ac6ba79e37b427796fefbb7b85f5ead [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f27a46e93be4dacb46c0acd7adcc744" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43421 } } peers { permanent_uuid: "8eb12ff1288c41888d3e54ae0fdbcf78" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41853 } } peers { permanent_uuid: "3ac6ba79e37b427796fefbb7b85f5ead" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33711 } }
I20260928 05:22:01.134853 20701 raft_consensus.cc:348] T 09c79774d5c64b85aa685f1d606846bc P 8eb12ff1288c41888d3e54ae0fdbcf78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f27a46e93be4dacb46c0acd7adcc744" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43421 } } peers { permanent_uuid: "8eb12ff1288c41888d3e54ae0fdbcf78" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41853 } } peers { permanent_uuid: "3ac6ba79e37b427796fefbb7b85f5ead" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33711 } }
I20260928 05:22:01.135245 20701 raft_consensus.cc:374] T 09c79774d5c64b85aa685f1d606846bc P 8eb12ff1288c41888d3e54ae0fdbcf78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:22:01.135339 20701 raft_consensus.cc:729] T 09c79774d5c64b85aa685f1d606846bc P 8eb12ff1288c41888d3e54ae0fdbcf78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8eb12ff1288c41888d3e54ae0fdbcf78, State: Initialized, Role: FOLLOWER
I20260928 05:22:01.135648 20551 heartbeater.cc:503] Master 127.0.97.62:34487 was elected leader, sending a full tablet report...
I20260928 05:22:01.135861 20701 consensus_queue.cc:260] T 09c79774d5c64b85aa685f1d606846bc P 8eb12ff1288c41888d3e54ae0fdbcf78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f27a46e93be4dacb46c0acd7adcc744" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43421 } } peers { permanent_uuid: "8eb12ff1288c41888d3e54ae0fdbcf78" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41853 } } peers { permanent_uuid: "3ac6ba79e37b427796fefbb7b85f5ead" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33711 } }
I20260928 05:22:01.136319 20700 ts_tablet_manager.cc:1428] T 09c79774d5c64b85aa685f1d606846bc P 3ac6ba79e37b427796fefbb7b85f5ead: Time spent starting tablet: real 0.013s user 0.011s sys 0.002s
I20260928 05:22:01.136935 20421 heartbeater.cc:503] Master 127.0.97.62:34487 was elected leader, sending a full tablet report...
I20260928 05:22:01.137497 20701 ts_tablet_manager.cc:1428] T 09c79774d5c64b85aa685f1d606846bc P 8eb12ff1288c41888d3e54ae0fdbcf78: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
W20260928 05:22:01.164968 20422 tablet.cc:2369] T 09c79774d5c64b85aa685f1d606846bc P 8eb12ff1288c41888d3e54ae0fdbcf78: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 05:22:01.201241 20707 raft_consensus.cc:482] T 09c79774d5c64b85aa685f1d606846bc P 8eb12ff1288c41888d3e54ae0fdbcf78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:22:01.201550 20707 raft_consensus.cc:504] T 09c79774d5c64b85aa685f1d606846bc P 8eb12ff1288c41888d3e54ae0fdbcf78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f27a46e93be4dacb46c0acd7adcc744" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43421 } } peers { permanent_uuid: "8eb12ff1288c41888d3e54ae0fdbcf78" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41853 } } peers { permanent_uuid: "3ac6ba79e37b427796fefbb7b85f5ead" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33711 } }
I20260928 05:22:01.203176 20707 leader_election.cc:288] T 09c79774d5c64b85aa685f1d606846bc P 8eb12ff1288c41888d3e54ae0fdbcf78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f27a46e93be4dacb46c0acd7adcc744 (127.0.97.3:43421), 3ac6ba79e37b427796fefbb7b85f5ead (127.0.97.2:33711)
I20260928 05:22:01.209873 20708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:01.203473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33711 (local address 127.0.97.1:41805)
0928 05:22:01.204146 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:01.204251 (+ 105us) client_negotiation.cc:174] Beginning negotiation
0928 05:22:01.204431 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:22:01.204797 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:22:01.204801 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:22:01.204828 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:22:01.205251 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:01.205264 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:01.206258 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:01.206263 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:01.207877 (+ 1614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:01.207892 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:01.209182 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:01.209188 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:01.209346 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:01.209380 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 05:22:01.209470 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 05:22:01.209577 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"mutex_wait_us":63,"thread_start_us":373,"threads_started":1}
I20260928 05:22:01.209950 20696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:01.203705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41805 (local address 127.0.97.2:33711)
0928 05:22:01.203987 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:01.203994 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:01.204367 (+ 373us) server_negotiation.cc:407] Connection header received
0928 05:22:01.204591 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:01.204595 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:01.204669 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:01.204745 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:22:01.205397 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:01.206123 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:01.208063 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:01.208986 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:01.209099 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:01.209180 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:01.209532 (+ 352us) server_negotiation.cc:299] Negotiation successful
0928 05:22:01.209677 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135}
I20260928 05:22:01.210153 20694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:01.203650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48281 (local address 127.0.97.3:43421)
0928 05:22:01.203942 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:01.203949 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:01.204571 (+ 622us) server_negotiation.cc:407] Connection header received
0928 05:22:01.204723 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:01.204729 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:01.204804 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:01.204938 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:22:01.205507 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:01.206230 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:01.208337 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:01.209240 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:01.209356 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:01.209443 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:01.209787 (+ 344us) server_negotiation.cc:299] Negotiation successful
0928 05:22:01.209913 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141}
I20260928 05:22:01.210201 20709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:01.203527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43421 (local address 127.0.97.1:48281)
0928 05:22:01.204466 (+ 939us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:01.204479 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 05:22:01.204600 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:22:01.204958 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:22:01.204964 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:22:01.204993 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:22:01.205338 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:01.205349 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:01.206388 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:01.206392 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:01.208136 (+ 1744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:01.208151 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:01.209417 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:01.209422 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:01.209604 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:01.209642 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 05:22:01.209739 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 05:22:01.209867 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":831,"thread_start_us":683,"threads_started":1}
I20260928 05:22:01.211103 20636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09c79774d5c64b85aa685f1d606846bc" candidate_uuid: "8eb12ff1288c41888d3e54ae0fdbcf78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f27a46e93be4dacb46c0acd7adcc744" is_pre_election: true
I20260928 05:22:01.211179 20506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09c79774d5c64b85aa685f1d606846bc" candidate_uuid: "8eb12ff1288c41888d3e54ae0fdbcf78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ac6ba79e37b427796fefbb7b85f5ead" is_pre_election: true
I20260928 05:22:01.211644 20636 raft_consensus.cc:2454] T 09c79774d5c64b85aa685f1d606846bc P 1f27a46e93be4dacb46c0acd7adcc744 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8eb12ff1288c41888d3e54ae0fdbcf78 in term 0.
I20260928 05:22:01.211691 20506 raft_consensus.cc:2454] T 09c79774d5c64b85aa685f1d606846bc P 3ac6ba79e37b427796fefbb7b85f5ead [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8eb12ff1288c41888d3e54ae0fdbcf78 in term 0.
I20260928 05:22:01.212253 20309 leader_election.cc:302] T 09c79774d5c64b85aa685f1d606846bc P 8eb12ff1288c41888d3e54ae0fdbcf78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ac6ba79e37b427796fefbb7b85f5ead, 8eb12ff1288c41888d3e54ae0fdbcf78; no voters:
I20260928 05:22:01.212757 20707 raft_consensus.cc:2781] T 09c79774d5c64b85aa685f1d606846bc P 8eb12ff1288c41888d3e54ae0fdbcf78 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 05:22:01.212918 20707 raft_consensus.cc:482] T 09c79774d5c64b85aa685f1d606846bc P 8eb12ff1288c41888d3e54ae0fdbcf78 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 05:22:01.213034 20707 raft_consensus.cc:3037] T 09c79774d5c64b85aa685f1d606846bc P 8eb12ff1288c41888d3e54ae0fdbcf78 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:22:01.215323 20707 raft_consensus.cc:504] T 09c79774d5c64b85aa685f1d606846bc P 8eb12ff1288c41888d3e54ae0fdbcf78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f27a46e93be4dacb46c0acd7adcc744" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43421 } } peers { permanent_uuid: "8eb12ff1288c41888d3e54ae0fdbcf78" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41853 } } peers { permanent_uuid: "3ac6ba79e37b427796fefbb7b85f5ead" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33711 } }
I20260928 05:22:01.215991 20707 leader_election.cc:288] T 09c79774d5c64b85aa685f1d606846bc P 8eb12ff1288c41888d3e54ae0fdbcf78 [CANDIDATE]: Term 1 election: Requested vote from peers 1f27a46e93be4dacb46c0acd7adcc744 (127.0.97.3:43421), 3ac6ba79e37b427796fefbb7b85f5ead (127.0.97.2:33711)
I20260928 05:22:01.216555 20506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09c79774d5c64b85aa685f1d606846bc" candidate_uuid: "8eb12ff1288c41888d3e54ae0fdbcf78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ac6ba79e37b427796fefbb7b85f5ead"
I20260928 05:22:01.216570 20636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09c79774d5c64b85aa685f1d606846bc" candidate_uuid: "8eb12ff1288c41888d3e54ae0fdbcf78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f27a46e93be4dacb46c0acd7adcc744"
I20260928 05:22:01.216729 20636 raft_consensus.cc:3037] T 09c79774d5c64b85aa685f1d606846bc P 1f27a46e93be4dacb46c0acd7adcc744 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:22:01.216763 20506 raft_consensus.cc:3037] T 09c79774d5c64b85aa685f1d606846bc P 3ac6ba79e37b427796fefbb7b85f5ead [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:22:01.218915 20636 raft_consensus.cc:2454] T 09c79774d5c64b85aa685f1d606846bc P 1f27a46e93be4dacb46c0acd7adcc744 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8eb12ff1288c41888d3e54ae0fdbcf78 in term 1.
I20260928 05:22:01.219018 20506 raft_consensus.cc:2454] T 09c79774d5c64b85aa685f1d606846bc P 3ac6ba79e37b427796fefbb7b85f5ead [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8eb12ff1288c41888d3e54ae0fdbcf78 in term 1.
I20260928 05:22:01.219342 20310 leader_election.cc:302] T 09c79774d5c64b85aa685f1d606846bc P 8eb12ff1288c41888d3e54ae0fdbcf78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f27a46e93be4dacb46c0acd7adcc744, 8eb12ff1288c41888d3e54ae0fdbcf78; no voters:
I20260928 05:22:01.219617 20707 raft_consensus.cc:2781] T 09c79774d5c64b85aa685f1d606846bc P 8eb12ff1288c41888d3e54ae0fdbcf78 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:22:01.220023 20707 raft_consensus.cc:686] T 09c79774d5c64b85aa685f1d606846bc P 8eb12ff1288c41888d3e54ae0fdbcf78 [term 1 LEADER]: Becoming Leader. State: Replica: 8eb12ff1288c41888d3e54ae0fdbcf78, State: Running, Role: LEADER
I20260928 05:22:01.220463 20707 consensus_queue.cc:237] T 09c79774d5c64b85aa685f1d606846bc P 8eb12ff1288c41888d3e54ae0fdbcf78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f27a46e93be4dacb46c0acd7adcc744" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43421 } } peers { permanent_uuid: "8eb12ff1288c41888d3e54ae0fdbcf78" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41853 } } peers { permanent_uuid: "3ac6ba79e37b427796fefbb7b85f5ead" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33711 } }
I20260928 05:22:01.224730 20234 catalog_manager.cc:5495] T 09c79774d5c64b85aa685f1d606846bc P 8eb12ff1288c41888d3e54ae0fdbcf78 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8eb12ff1288c41888d3e54ae0fdbcf78 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "8eb12ff1288c41888d3e54ae0fdbcf78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f27a46e93be4dacb46c0acd7adcc744" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8eb12ff1288c41888d3e54ae0fdbcf78" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3ac6ba79e37b427796fefbb7b85f5ead" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33711 } health_report { overall_health: UNKNOWN } } }
I20260928 05:22:01.241622 20698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:01.236615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37158 (local address 127.0.97.1:41853)
0928 05:22:01.236783 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:01.236788 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:01.236804 (+ 16us) server_negotiation.cc:407] Connection header received
0928 05:22:01.237016 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:01.237019 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:01.237110 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:01.237231 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:22:01.237613 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:01.238243 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:01.239598 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:01.239979 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:01.240071 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:01.240602 (+ 531us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:22:01.241122 (+ 520us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:22:01.241249 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:01.241342 (+ 93us) server_negotiation.cc:299] Negotiation successful
0928 05:22:01.241400 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 05:22:01.249306 20636 raft_consensus.cc:1261] T 09c79774d5c64b85aa685f1d606846bc P 1f27a46e93be4dacb46c0acd7adcc744 [term 1 FOLLOWER]: Refusing update from remote peer 8eb12ff1288c41888d3e54ae0fdbcf78: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:22:01.249306 20506 raft_consensus.cc:1261] T 09c79774d5c64b85aa685f1d606846bc P 3ac6ba79e37b427796fefbb7b85f5ead [term 1 FOLLOWER]: Refusing update from remote peer 8eb12ff1288c41888d3e54ae0fdbcf78: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:22:01.250346 20710 consensus_queue.cc:1035] T 09c79774d5c64b85aa685f1d606846bc P 8eb12ff1288c41888d3e54ae0fdbcf78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ac6ba79e37b427796fefbb7b85f5ead" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:22:01.250700 20707 consensus_queue.cc:1035] T 09c79774d5c64b85aa685f1d606846bc P 8eb12ff1288c41888d3e54ae0fdbcf78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f27a46e93be4dacb46c0acd7adcc744" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:22:01.260979 20713 mvcc.cc:205] Tried to move back new op lower bound from 7334186685427511296 to 7334186685322772480. Current Snapshot: MvccSnapshot[applied={T|T < 7334186685427511296}]
I20260928 05:22:01.262658 20714 mvcc.cc:205] Tried to move back new op lower bound from 7334186685427511296 to 7334186685322772480. Current Snapshot: MvccSnapshot[applied={T|T < 7334186685427511296}]
I20260928 05:22:01.272877 20235 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58072:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:22:01.273168 20235 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58072:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:22:01.276597 20235 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d09294f1e77c410aa1fb302657f73df0: Sending DeleteTablet for 3 replicas of tablet 09c79774d5c64b85aa685f1d606846bc
I20260928 05:22:01.277709 20356 tablet_service.cc:1514] Processing DeleteTablet for tablet 09c79774d5c64b85aa685f1d606846bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:22:01 UTC) from {username='slave'} at 127.0.0.1:37150
I20260928 05:22:01.277705 20486 tablet_service.cc:1514] Processing DeleteTablet for tablet 09c79774d5c64b85aa685f1d606846bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:22:01 UTC) from {username='slave'} at 127.0.0.1:49078
I20260928 05:22:01.278684 20724 tablet_replica.cc:330] T 09c79774d5c64b85aa685f1d606846bc P 3ac6ba79e37b427796fefbb7b85f5ead: stopping tablet replica
I20260928 05:22:01.278820 20616 tablet_service.cc:1514] Processing DeleteTablet for tablet 09c79774d5c64b85aa685f1d606846bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:22:01 UTC) from {username='slave'} at 127.0.0.1:58318
I20260928 05:22:01.278913 20723 tablet_replica.cc:330] T 09c79774d5c64b85aa685f1d606846bc P 8eb12ff1288c41888d3e54ae0fdbcf78: stopping tablet replica
I20260928 05:22:01.279093 20724 raft_consensus.cc:2229] T 09c79774d5c64b85aa685f1d606846bc P 3ac6ba79e37b427796fefbb7b85f5ead [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:22:01.279316 20723 raft_consensus.cc:2229] T 09c79774d5c64b85aa685f1d606846bc P 8eb12ff1288c41888d3e54ae0fdbcf78 [term 1 LEADER]: Raft consensus shutting down.
I20260928 05:22:01.279564 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 20294
I20260928 05:22:01.279543 20724 raft_consensus.cc:2258] T 09c79774d5c64b85aa685f1d606846bc P 3ac6ba79e37b427796fefbb7b85f5ead [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:22:01.280010 20723 raft_consensus.cc:2258] T 09c79774d5c64b85aa685f1d606846bc P 8eb12ff1288c41888d3e54ae0fdbcf78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:22:01.281936 20724 ts_tablet_manager.cc:1905] T 09c79774d5c64b85aa685f1d606846bc P 3ac6ba79e37b427796fefbb7b85f5ead: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:22:01.281950 20723 ts_tablet_manager.cc:1905] T 09c79774d5c64b85aa685f1d606846bc P 8eb12ff1288c41888d3e54ae0fdbcf78: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:22:01.288560 20724 ts_tablet_manager.cc:1918] T 09c79774d5c64b85aa685f1d606846bc P 3ac6ba79e37b427796fefbb7b85f5ead: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:22:01.288721 20724 log.cc:1196] T 09c79774d5c64b85aa685f1d606846bc P 3ac6ba79e37b427796fefbb7b85f5ead: Deleting WAL directory at /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572622576160-388-0/minicluster-data/ts-1/wal/wals/09c79774d5c64b85aa685f1d606846bc
I20260928 05:22:01.289176 20725 tablet_replica.cc:330] T 09c79774d5c64b85aa685f1d606846bc P 1f27a46e93be4dacb46c0acd7adcc744: stopping tablet replica
I20260928 05:22:01.289251 20724 ts_tablet_manager.cc:1939] T 09c79774d5c64b85aa685f1d606846bc P 3ac6ba79e37b427796fefbb7b85f5ead: Deleting consensus metadata
I20260928 05:22:01.289646 20725 raft_consensus.cc:2229] T 09c79774d5c64b85aa685f1d606846bc P 1f27a46e93be4dacb46c0acd7adcc744 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:22:01.290143 20725 raft_consensus.cc:2258] T 09c79774d5c64b85aa685f1d606846bc P 1f27a46e93be4dacb46c0acd7adcc744 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:22:01.291170 20219 catalog_manager.cc:4841] TS 3ac6ba79e37b427796fefbb7b85f5ead (127.0.97.2:33711): tablet 09c79774d5c64b85aa685f1d606846bc (table test-table [id=b9b78ed48914429087c5cd7fd56e1f18]) successfully deleted
I20260928 05:22:01.292078 20725 ts_tablet_manager.cc:1905] T 09c79774d5c64b85aa685f1d606846bc P 1f27a46e93be4dacb46c0acd7adcc744: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:22:01.297201 20725 ts_tablet_manager.cc:1918] T 09c79774d5c64b85aa685f1d606846bc P 1f27a46e93be4dacb46c0acd7adcc744: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:22:01.297324 20725 log.cc:1196] T 09c79774d5c64b85aa685f1d606846bc P 1f27a46e93be4dacb46c0acd7adcc744: Deleting WAL directory at /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572622576160-388-0/minicluster-data/ts-2/wal/wals/09c79774d5c64b85aa685f1d606846bc
I20260928 05:22:01.297729 20725 ts_tablet_manager.cc:1939] T 09c79774d5c64b85aa685f1d606846bc P 1f27a46e93be4dacb46c0acd7adcc744: Deleting consensus metadata
I20260928 05:22:01.299044 20723 ts_tablet_manager.cc:1918] T 09c79774d5c64b85aa685f1d606846bc P 8eb12ff1288c41888d3e54ae0fdbcf78: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:22:01.299222 20723 log.cc:1196] T 09c79774d5c64b85aa685f1d606846bc P 8eb12ff1288c41888d3e54ae0fdbcf78: Deleting WAL directory at /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572622576160-388-0/minicluster-data/ts-0/wal/wals/09c79774d5c64b85aa685f1d606846bc
I20260928 05:22:01.299242 20220 catalog_manager.cc:4841] TS 1f27a46e93be4dacb46c0acd7adcc744 (127.0.97.3:43421): tablet 09c79774d5c64b85aa685f1d606846bc (table test-table [id=b9b78ed48914429087c5cd7fd56e1f18]) successfully deleted
I20260928 05:22:01.299687 20723 ts_tablet_manager.cc:1939] T 09c79774d5c64b85aa685f1d606846bc P 8eb12ff1288c41888d3e54ae0fdbcf78: Deleting consensus metadata
I20260928 05:22:01.301122 20220 catalog_manager.cc:4841] TS 8eb12ff1288c41888d3e54ae0fdbcf78 (127.0.97.1:41853): tablet 09c79774d5c64b85aa685f1d606846bc (table test-table [id=b9b78ed48914429087c5cd7fd56e1f18]) successfully deleted
I20260928 05:22:01.302241 20698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:01.281978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37162 (local address 127.0.97.1:41853)
0928 05:22:01.282144 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:01.282151 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:01.294508 (+ 12357us) server_negotiation.cc:407] Connection header received
0928 05:22:01.294730 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:01.294736 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:01.294833 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:01.294948 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:01.295440 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:01.296575 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:01.298501 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:01.299112 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:01.299237 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:01.300758 (+ 1521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:01.300768 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:01.300784 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:22:01.301517 (+ 733us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:01.301560 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:22:01.301580 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:01.301667 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:01.301838 (+ 171us) server_negotiation.cc:299] Negotiation successful
0928 05:22:01.301910 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 05:22:01.302815 20416 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:22:01.396893 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 20294
I20260928 05:22:01.411697 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 20424
I20260928 05:22:01.418015 20696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:01.412470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49094 (local address 127.0.97.2:33711)
0928 05:22:01.412650 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:01.412655 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:01.412672 (+ 17us) server_negotiation.cc:407] Connection header received
0928 05:22:01.412760 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:01.412764 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:01.412840 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:01.412963 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:01.413349 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:01.414115 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:01.415283 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:01.415663 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:01.415751 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:01.416701 (+ 950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:01.416709 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:01.416723 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:22:01.417350 (+ 627us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:01.417396 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:22:01.417416 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:01.417498 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:01.417610 (+ 112us) server_negotiation.cc:299] Negotiation successful
0928 05:22:01.417707 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 05:22:01.418654 20546 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:22:01.504231 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 20424
I20260928 05:22:01.517364 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 20554
I20260928 05:22:01.523685 20694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:01.518070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58334 (local address 127.0.97.3:43421)
0928 05:22:01.518259 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:01.518264 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:01.518282 (+ 18us) server_negotiation.cc:407] Connection header received
0928 05:22:01.518392 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:01.518395 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:01.518476 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:01.518566 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:01.518918 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:01.519632 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:01.520834 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:01.521320 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:01.521436 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:01.522345 (+ 909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:01.522354 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:01.522368 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:22:01.522970 (+ 602us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:01.523022 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:22:01.523044 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:01.523138 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:01.523230 (+ 92us) server_negotiation.cc:299] Negotiation successful
0928 05:22:01.523329 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 05:22:01.524313 20676 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:22:01.610116 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 20554
I20260928 05:22:01.623075 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 20204
I20260928 05:22:01.624195 20265 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:22:01.729537 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 20204
2026-09-28T05:22:01Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2547 ms)
[----------] 6 tests from EncryptionPolicyTest (19656 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790572622576160-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:22:01 dist-test-slave-whsn krb5kdc[20733](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:22:01 dist-test-slave-whsn krb5kdc[20733](info): set up 2 sockets
Sep 28 05:22:01 dist-test-slave-whsn krb5kdc[20733](info): commencing operation
krb5kdc: starting...
W20260928 05:22:02.737051 388 mini_kdc.cc:121] Time spent starting KDC: real 0.971s user 0.005s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790572622576160-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790572622576160-388-0/krb5kdc/test-user.keytab.
Sep 28 05:22:02 dist-test-slave-whsn krb5kdc[20733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572922, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:22:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:22:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790572622576160-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790572622576160-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 05:22:02.952831 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790572622576160-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790572622576160-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790572622576160-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33735
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790572622576160-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33735
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:22:03.259908 20746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:22:03.260175 20746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:22:03.260305 20746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:22:03.267313 20746 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:22:03.267416 20746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:22:03.267485 20746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:22:03.267565 20746 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:22:03.267621 20746 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:22:03.277006 20746 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790572622576160-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790572622576160-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33735
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790572622576160-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33735
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790572622576160-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:22:03.278342 20746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:22:03.279606 20746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:22:03.290256 20751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:22:03.290292 20752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:22:03.290337 20754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:22:03.291316 20746 server_base.cc:1034] running on GCE node
I20260928 05:22:03.291978 20746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:22:03.293258 20746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:22:03.294448 20746 hybrid_clock.cc:648] HybridClock initialized: now 1790572923294409 us; error 33 us; skew 500 ppm
I20260928 05:22:03.294747 20746 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:22:03 dist-test-slave-whsn krb5kdc[20733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572923, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:22:03.297596 20746 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 05:22:03.298920 20746 webserver.cc:460] Webserver started at http://127.0.0.1:42775/ using document root <none> and password file <none>
I20260928 05:22:03.299503 20746 fs_manager.cc:359] Metadata directory not provided
I20260928 05:22:03.299594 20746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:22:03.299820 20746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:22:03.302552 20746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790572622576160-388-0/minicluster-data/master-0/data/instance:
uuid: "e6c86ace4e274078892918b97970a20d"
format_stamp: "Formatted at 2026-09-28 05:22:03 on dist-test-slave-whsn"
server_key: "e7395fa3ae43d4c4eca68a1a1d3106e5"
server_key_iv: "d05442c86e2b469d9ebf12af0ba7a0db"
server_key_version: "encryptionkey@0"
I20260928 05:22:03.303220 20746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790572622576160-388-0/minicluster-data/master-0/wal/instance:
uuid: "e6c86ace4e274078892918b97970a20d"
format_stamp: "Formatted at 2026-09-28 05:22:03 on dist-test-slave-whsn"
server_key: "e7395fa3ae43d4c4eca68a1a1d3106e5"
server_key_iv: "d05442c86e2b469d9ebf12af0ba7a0db"
server_key_version: "encryptionkey@0"
I20260928 05:22:03.307961 20746 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 05:22:03.311138 20761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:22:03.312364 20746 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 05:22:03.312505 20746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790572622576160-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790572622576160-388-0/minicluster-data/master-0/wal
uuid: "e6c86ace4e274078892918b97970a20d"
format_stamp: "Formatted at 2026-09-28 05:22:03 on dist-test-slave-whsn"
server_key: "e7395fa3ae43d4c4eca68a1a1d3106e5"
server_key_iv: "d05442c86e2b469d9ebf12af0ba7a0db"
server_key_version: "encryptionkey@0"
I20260928 05:22:03.312670 20746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790572622576160-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790572622576160-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790572622576160-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:22:03.324455 20746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:22:03.328074 20746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:22:03.328310 20746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:22:03.344703 20746 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33735
I20260928 05:22:03.344738 20812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33735 every 8 connection(s)
I20260928 05:22:03.345935 20746 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:22:03.349287 20813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:22:03.351442 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 20746
I20260928 05:22:03.351653 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790572622576160-388-0/minicluster-data/master-0/wal/instance
I20260928 05:22:03.352375 388 external_mini_cluster.cc:1428] Setting key cd1375898469feeec68ca030371b2ccf
I20260928 05:22:03.360126 20813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6c86ace4e274078892918b97970a20d: Bootstrap starting.
I20260928 05:22:03.363032 20813 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e6c86ace4e274078892918b97970a20d: Neither blocks nor log segments found. Creating new log.
I20260928 05:22:03.363922 20813 log.cc:824] T 00000000000000000000000000000000 P e6c86ace4e274078892918b97970a20d: Log is configured to *not* fsync() on all Append() calls
Sep 28 05:22:03 dist-test-slave-whsn krb5kdc[20733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572922, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 05:22:03.366585 20813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6c86ace4e274078892918b97970a20d: No bootstrap required, opened a new log
I20260928 05:22:03.373034 20816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:03.354176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36983 (local address 127.0.0.1:33735)
0928 05:22:03.354905 (+ 729us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:03.354916 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:03.354955 (+ 39us) server_negotiation.cc:407] Connection header received
0928 05:22:03.356557 (+ 1602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:03.356579 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:03.356971 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:03.357300 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:03.358684 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:03.359929 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:03.361318 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:03.361963 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:03.362113 (+ 150us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:03.366146 (+ 4033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:03.366155 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:03.366168 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:22:03.366200 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:03.369186 (+ 2986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:03.369895 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:03.369901 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:03.369906 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:03.370059 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:03.370472 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:03.370475 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:03.370477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:03.370963 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:03.371294 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:03.371748 (+ 454us) server_negotiation.cc:299] Negotiation successful
0928 05:22:03.372047 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":318,"threads_started":1}
I20260928 05:22:03.374198 20813 raft_consensus.cc:348] T 00000000000000000000000000000000 P e6c86ace4e274078892918b97970a20d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6c86ace4e274078892918b97970a20d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33735 } }
I20260928 05:22:03.374516 20813 raft_consensus.cc:374] T 00000000000000000000000000000000 P e6c86ace4e274078892918b97970a20d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:22:03.374619 20813 raft_consensus.cc:729] T 00000000000000000000000000000000 P e6c86ace4e274078892918b97970a20d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6c86ace4e274078892918b97970a20d, State: Initialized, Role: FOLLOWER
I20260928 05:22:03.375138 20813 consensus_queue.cc:260] T 00000000000000000000000000000000 P e6c86ace4e274078892918b97970a20d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6c86ace4e274078892918b97970a20d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33735 } }
I20260928 05:22:03.375432 20813 raft_consensus.cc:388] T 00000000000000000000000000000000 P e6c86ace4e274078892918b97970a20d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:22:03.375554 20813 raft_consensus.cc:482] T 00000000000000000000000000000000 P e6c86ace4e274078892918b97970a20d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:22:03.375700 20813 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e6c86ace4e274078892918b97970a20d [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:22:03.376760 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 05:22:03.377924 20813 raft_consensus.cc:504] T 00000000000000000000000000000000 P e6c86ace4e274078892918b97970a20d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6c86ace4e274078892918b97970a20d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33735 } }
I20260928 05:22:03.378350 20813 leader_election.cc:302] T 00000000000000000000000000000000 P e6c86ace4e274078892918b97970a20d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e6c86ace4e274078892918b97970a20d; no voters:
I20260928 05:22:03.378872 20813 leader_election.cc:288] T 00000000000000000000000000000000 P e6c86ace4e274078892918b97970a20d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:22:03.379045 20818 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e6c86ace4e274078892918b97970a20d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:22:03.380555 20813 sys_catalog.cc:564] T 00000000000000000000000000000000 P e6c86ace4e274078892918b97970a20d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:22:03.385187 20818 raft_consensus.cc:686] T 00000000000000000000000000000000 P e6c86ace4e274078892918b97970a20d [term 1 LEADER]: Becoming Leader. State: Replica: e6c86ace4e274078892918b97970a20d, State: Running, Role: LEADER
I20260928 05:22:03.385795 20818 consensus_queue.cc:237] T 00000000000000000000000000000000 P e6c86ace4e274078892918b97970a20d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6c86ace4e274078892918b97970a20d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33735 } }
I20260928 05:22:03.388705 20821 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6c86ace4e274078892918b97970a20d [sys.catalog]: SysCatalogTable state changed. Reason: New leader e6c86ace4e274078892918b97970a20d. Latest consensus state: current_term: 1 leader_uuid: "e6c86ace4e274078892918b97970a20d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6c86ace4e274078892918b97970a20d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33735 } } }
I20260928 05:22:03.389320 20819 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6c86ace4e274078892918b97970a20d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e6c86ace4e274078892918b97970a20d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6c86ace4e274078892918b97970a20d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33735 } } }
I20260928 05:22:03.389761 20819 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6c86ace4e274078892918b97970a20d [sys.catalog]: This master's current role is: LEADER
I20260928 05:22:03.389618 20821 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6c86ace4e274078892918b97970a20d [sys.catalog]: This master's current role is: LEADER
W20260928 05:22:03.395879 20836 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e6c86ace4e274078892918b97970a20d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 05:22:03.396080 20836 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 05:22:03.396220 20837 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:22:03.398288 20837 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:22:03.407305 20837 catalog_manager.cc:1347] Generated new cluster ID: 6f31425452654e79bd91f19d41c09cff
I20260928 05:22:03.407487 20837 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:22:03.409909 20816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:03.391458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47731 (local address 127.0.0.1:33735)
0928 05:22:03.391644 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:03.391652 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:03.393595 (+ 1943us) server_negotiation.cc:407] Connection header received
0928 05:22:03.393748 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:03.393753 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:03.393846 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:03.393978 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:03.395988 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:03.397316 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:03.399320 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:03.399999 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:03.400145 (+ 146us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:03.403314 (+ 3169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:03.403321 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:03.403326 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:22:03.403361 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:03.406400 (+ 3039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:03.407550 (+ 1150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:03.407555 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:03.407559 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:03.407722 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:03.408280 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:03.408286 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:03.408289 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:03.408680 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:03.408961 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:03.409389 (+ 428us) server_negotiation.cc:299] Negotiation successful
0928 05:22:03.409571 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 05:22:03.433429 20837 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:22:03.435269 20837 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:22:03.448622 20837 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e6c86ace4e274078892918b97970a20d: Generated new TSK 0
I20260928 05:22:03.449441 20837 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 05:22:03.460728 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 20746
I20260928 05:22:03.461985 20808 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:22:03.553737 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 20746
W20260928 05:22:03.568141 20731 connection.cc:535] client connection to 127.0.0.1:33735 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33735): Connection reset by peer (error 104)
2026-09-28T05:22:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1820 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790572622576160-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:22:03 dist-test-slave-whsn krb5kdc[20845](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:22:03 dist-test-slave-whsn krb5kdc[20845](info): set up 2 sockets
Sep 28 05:22:03 dist-test-slave-whsn krb5kdc[20845](info): commencing operation
krb5kdc: starting...
W20260928 05:22:04.572834 388 mini_kdc.cc:121] Time spent starting KDC: real 0.986s user 0.005s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790572622576160-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790572622576160-388-0/krb5kdc/test-user.keytab.
Sep 28 05:22:04 dist-test-slave-whsn krb5kdc[20845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:22:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:22:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790572622576160-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790572622576160-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 05:22:04.783991 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790572622576160-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790572622576160-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790572622576160-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39485
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790572622576160-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39485
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:22:05.088420 20858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:22:05.088688 20858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:22:05.088819 20858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:22:05.095814 20858 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:22:05.095911 20858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:22:05.095988 20858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:22:05.096082 20858 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:22:05.096139 20858 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:22:05.105522 20858 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790572622576160-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790572622576160-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39485
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790572622576160-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790572622576160-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:22:05.106662 20858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:22:05.107864 20858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:22:05.118609 20864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:22:05.118649 20866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:22:05.118903 20863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:22:05.119427 20858 server_base.cc:1034] running on GCE node
I20260928 05:22:05.120110 20858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:22:05.121393 20858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:22:05.122578 20858 hybrid_clock.cc:648] HybridClock initialized: now 1790572925122530 us; error 43 us; skew 500 ppm
I20260928 05:22:05.122869 20858 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:22:05 dist-test-slave-whsn krb5kdc[20845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572925, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:22:05.125689 20858 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 05:22:05.127269 20858 webserver.cc:460] Webserver started at http://127.0.0.1:37281/ using document root <none> and password file <none>
I20260928 05:22:05.128083 20858 fs_manager.cc:359] Metadata directory not provided
I20260928 05:22:05.128211 20858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:22:05.128485 20858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:22:05.131397 20858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790572622576160-388-0/minicluster-data/master-0/data/instance:
uuid: "36427ce5899a46d6b8a7ac74829c3e70"
format_stamp: "Formatted at 2026-09-28 05:22:05 on dist-test-slave-whsn"
server_key: "b6f9e4e0a208497c7cd9cd6105c514d9"
server_key_iv: "c23056c457c9174be55b1ad4cb3637c4"
server_key_version: "encryptionkey@0"
I20260928 05:22:05.132092 20858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790572622576160-388-0/minicluster-data/master-0/wal/instance:
uuid: "36427ce5899a46d6b8a7ac74829c3e70"
format_stamp: "Formatted at 2026-09-28 05:22:05 on dist-test-slave-whsn"
server_key: "b6f9e4e0a208497c7cd9cd6105c514d9"
server_key_iv: "c23056c457c9174be55b1ad4cb3637c4"
server_key_version: "encryptionkey@0"
I20260928 05:22:05.136737 20858 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 05:22:05.139842 20873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:22:05.141064 20858 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 05:22:05.141212 20858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790572622576160-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790572622576160-388-0/minicluster-data/master-0/wal
uuid: "36427ce5899a46d6b8a7ac74829c3e70"
format_stamp: "Formatted at 2026-09-28 05:22:05 on dist-test-slave-whsn"
server_key: "b6f9e4e0a208497c7cd9cd6105c514d9"
server_key_iv: "c23056c457c9174be55b1ad4cb3637c4"
server_key_version: "encryptionkey@0"
I20260928 05:22:05.141371 20858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790572622576160-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790572622576160-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790572622576160-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:22:05.156419 20858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:22:05.160075 20858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:22:05.160312 20858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:22:05.176771 20858 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39485
I20260928 05:22:05.176801 20924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39485 every 8 connection(s)
I20260928 05:22:05.178032 20858 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:22:05.181432 20925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:22:05.181974 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 20858
I20260928 05:22:05.182156 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790572622576160-388-0/minicluster-data/master-0/wal/instance
I20260928 05:22:05.182699 388 external_mini_cluster.cc:1428] Setting key 9cd3ceca8822635656f3e74b2fef3ef3
I20260928 05:22:05.194235 20925 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36427ce5899a46d6b8a7ac74829c3e70: Bootstrap starting.
I20260928 05:22:05.195647 20928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:05.184305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40363 (local address 127.0.0.1:39485)
0928 05:22:05.185071 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:05.185082 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:05.185123 (+ 41us) server_negotiation.cc:407] Connection header received
0928 05:22:05.187216 (+ 2093us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:05.187248 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:05.187620 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:05.187940 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:05.188474 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:05.189732 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:05.191575 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:05.192233 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:05.192359 (+ 126us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:05.193554 (+ 1195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:05.193561 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:05.193574 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:22:05.194059 (+ 485us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:05.194136 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:22:05.194158 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:05.194286 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:05.194543 (+ 257us) server_negotiation.cc:299] Negotiation successful
0928 05:22:05.194734 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":336,"threads_started":1}
I20260928 05:22:05.197306 20925 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 36427ce5899a46d6b8a7ac74829c3e70: Neither blocks nor log segments found. Creating new log.
I20260928 05:22:05.198360 20925 log.cc:824] T 00000000000000000000000000000000 P 36427ce5899a46d6b8a7ac74829c3e70: Log is configured to *not* fsync() on all Append() calls
I20260928 05:22:05.199107 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 05:22:05.201217 20925 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36427ce5899a46d6b8a7ac74829c3e70: No bootstrap required, opened a new log
I20260928 05:22:05.209936 20928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:05.203735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46209 (local address 127.0.0.1:39485)
0928 05:22:05.203909 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:05.203914 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:05.204311 (+ 397us) server_negotiation.cc:407] Connection header received
0928 05:22:05.204433 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:05.204438 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:05.204506 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:05.204612 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:05.205050 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:05.206155 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:05.207459 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:05.208132 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:05.208268 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:05.209261 (+ 993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:05.209267 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:05.209271 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:22:05.209290 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:05.209332 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:22:05.209351 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:05.209465 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:05.209591 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 05:22:05.209637 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 05:22:05.210405 20925 raft_consensus.cc:348] T 00000000000000000000000000000000 P 36427ce5899a46d6b8a7ac74829c3e70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36427ce5899a46d6b8a7ac74829c3e70" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39485 } }
I20260928 05:22:05.210748 20925 raft_consensus.cc:374] T 00000000000000000000000000000000 P 36427ce5899a46d6b8a7ac74829c3e70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:22:05.210843 20925 raft_consensus.cc:729] T 00000000000000000000000000000000 P 36427ce5899a46d6b8a7ac74829c3e70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36427ce5899a46d6b8a7ac74829c3e70, State: Initialized, Role: FOLLOWER
I20260928 05:22:05.211344 20925 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36427ce5899a46d6b8a7ac74829c3e70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36427ce5899a46d6b8a7ac74829c3e70" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39485 } }
I20260928 05:22:05.211565 20925 raft_consensus.cc:388] T 00000000000000000000000000000000 P 36427ce5899a46d6b8a7ac74829c3e70 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:22:05.211683 20925 raft_consensus.cc:482] T 00000000000000000000000000000000 P 36427ce5899a46d6b8a7ac74829c3e70 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:22:05.211833 20925 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 36427ce5899a46d6b8a7ac74829c3e70 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:22:05.213907 20925 raft_consensus.cc:504] T 00000000000000000000000000000000 P 36427ce5899a46d6b8a7ac74829c3e70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36427ce5899a46d6b8a7ac74829c3e70" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39485 } }
I20260928 05:22:05.214337 20925 leader_election.cc:302] T 00000000000000000000000000000000 P 36427ce5899a46d6b8a7ac74829c3e70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 36427ce5899a46d6b8a7ac74829c3e70; no voters:
I20260928 05:22:05.214838 20925 leader_election.cc:288] T 00000000000000000000000000000000 P 36427ce5899a46d6b8a7ac74829c3e70 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:22:05.214931 20935 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 36427ce5899a46d6b8a7ac74829c3e70 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:22:05.215528 20935 raft_consensus.cc:686] T 00000000000000000000000000000000 P 36427ce5899a46d6b8a7ac74829c3e70 [term 1 LEADER]: Becoming Leader. State: Replica: 36427ce5899a46d6b8a7ac74829c3e70, State: Running, Role: LEADER
I20260928 05:22:05.215993 20935 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36427ce5899a46d6b8a7ac74829c3e70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36427ce5899a46d6b8a7ac74829c3e70" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39485 } }
I20260928 05:22:05.216626 20925 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36427ce5899a46d6b8a7ac74829c3e70 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:22:05.218605 20936 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36427ce5899a46d6b8a7ac74829c3e70 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "36427ce5899a46d6b8a7ac74829c3e70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36427ce5899a46d6b8a7ac74829c3e70" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39485 } } }
I20260928 05:22:05.218953 20936 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36427ce5899a46d6b8a7ac74829c3e70 [sys.catalog]: This master's current role is: LEADER
I20260928 05:22:05.219601 20937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36427ce5899a46d6b8a7ac74829c3e70 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36427ce5899a46d6b8a7ac74829c3e70. Latest consensus state: current_term: 1 leader_uuid: "36427ce5899a46d6b8a7ac74829c3e70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36427ce5899a46d6b8a7ac74829c3e70" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39485 } } }
I20260928 05:22:05.219924 20937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36427ce5899a46d6b8a7ac74829c3e70 [sys.catalog]: This master's current role is: LEADER
I20260928 05:22:05.226089 20943 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:22:05.230692 20943 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:22:05.238706 20943 catalog_manager.cc:1347] Generated new cluster ID: 9febaff7a3c94d5fbe27823aef5277c3
I20260928 05:22:05.238798 20943 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:22:05.255877 20943 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:22:05.257690 20943 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:22:05.283389 20943 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 36427ce5899a46d6b8a7ac74829c3e70: Generated new TSK 0
I20260928 05:22:05.284499 20943 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 05:22:05.298008 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 20858
W20260928 05:22:05.299631 20920 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:40363
W20260928 05:22:05.300465 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 20858: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 05:22:05.300673 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 20858
W20260928 05:22:05.313979 20843 connection.cc:535] client connection to 127.0.0.1:39485 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39485): Connection reset by peer (error 104)
2026-09-28T05:22:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1745 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790572622576160-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:22:05 dist-test-slave-whsn krb5kdc[20956](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:22:05 dist-test-slave-whsn krb5kdc[20956](info): set up 2 sockets
Sep 28 05:22:05 dist-test-slave-whsn krb5kdc[20956](info): commencing operation
krb5kdc: starting...
W20260928 05:22:06.305135 388 mini_kdc.cc:121] Time spent starting KDC: real 0.973s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790572622576160-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790572622576160-388-0/krb5kdc/test-user.keytab.
Sep 28 05:22:06 dist-test-slave-whsn krb5kdc[20956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:22:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:22:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790572622576160-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790572622576160-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 05:22:06.523924 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790572622576160-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790572622576160-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790572622576160-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33789
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790572622576160-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33789
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:22:06.829202 20969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:22:06.829473 20969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:22:06.829607 20969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:22:06.836429 20969 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:22:06.836532 20969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:22:06.836609 20969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:22:06.836689 20969 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:22:06.836750 20969 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:22:06.845835 20969 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790572622576160-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790572622576160-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33789
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790572622576160-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790572622576160-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:22:06.846967 20969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:22:06.848169 20969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:22:06.858935 20974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:22:06.858911 20975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:22:06.858879 20977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:22:06.859536 20969 server_base.cc:1034] running on GCE node
I20260928 05:22:06.860242 20969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:22:06.861550 20969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:22:06.862736 20969 hybrid_clock.cc:648] HybridClock initialized: now 1790572926862691 us; error 36 us; skew 500 ppm
I20260928 05:22:06.863036 20969 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:22:06 dist-test-slave-whsn krb5kdc[20956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572926, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:22:06.869210 20969 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 05:22:06.870690 20969 webserver.cc:460] Webserver started at http://127.0.0.1:40079/ using document root <none> and password file <none>
I20260928 05:22:06.871317 20969 fs_manager.cc:359] Metadata directory not provided
I20260928 05:22:06.871428 20969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:22:06.871697 20969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:22:06.874565 20969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790572622576160-388-0/minicluster-data/master-0/data/instance:
uuid: "5ed4026e0eb14d39a39f26939fe51365"
format_stamp: "Formatted at 2026-09-28 05:22:06 on dist-test-slave-whsn"
server_key: "9d9f8d5d6271747e132d855bc569880c"
server_key_iv: "d947afd610219565cc8f2bbc74c74d25"
server_key_version: "encryptionkey@0"
I20260928 05:22:06.875262 20969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790572622576160-388-0/minicluster-data/master-0/wal/instance:
uuid: "5ed4026e0eb14d39a39f26939fe51365"
format_stamp: "Formatted at 2026-09-28 05:22:06 on dist-test-slave-whsn"
server_key: "9d9f8d5d6271747e132d855bc569880c"
server_key_iv: "d947afd610219565cc8f2bbc74c74d25"
server_key_version: "encryptionkey@0"
I20260928 05:22:06.879948 20969 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260928 05:22:06.883214 20984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:22:06.884402 20969 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 05:22:06.884548 20969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790572622576160-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790572622576160-388-0/minicluster-data/master-0/wal
uuid: "5ed4026e0eb14d39a39f26939fe51365"
format_stamp: "Formatted at 2026-09-28 05:22:06 on dist-test-slave-whsn"
server_key: "9d9f8d5d6271747e132d855bc569880c"
server_key_iv: "d947afd610219565cc8f2bbc74c74d25"
server_key_version: "encryptionkey@0"
I20260928 05:22:06.884712 20969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790572622576160-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790572622576160-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790572622576160-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:22:06.890012 20969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:22:06.893841 20969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:22:06.894098 20969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:22:06.910259 20969 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33789
I20260928 05:22:06.910303 21035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33789 every 8 connection(s)
I20260928 05:22:06.911580 20969 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:22:06.915012 21036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:22:06.921593 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 20969
I20260928 05:22:06.921757 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790572622576160-388-0/minicluster-data/master-0/wal/instance
I20260928 05:22:06.922147 388 external_mini_cluster.cc:1428] Setting key b7b5a777485b5e543907af71ef43a226
I20260928 05:22:06.924690 21036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ed4026e0eb14d39a39f26939fe51365: Bootstrap starting.
I20260928 05:22:06.927891 21036 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ed4026e0eb14d39a39f26939fe51365: Neither blocks nor log segments found. Creating new log.
I20260928 05:22:06.928943 21036 log.cc:824] T 00000000000000000000000000000000 P 5ed4026e0eb14d39a39f26939fe51365: Log is configured to *not* fsync() on all Append() calls
I20260928 05:22:06.929129 21039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:06.923651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56835 (local address 127.0.0.1:33789)
0928 05:22:06.924351 (+ 700us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:06.924362 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:06.924404 (+ 42us) server_negotiation.cc:407] Connection header received
0928 05:22:06.926059 (+ 1655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:06.926082 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:06.926389 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:06.926688 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:06.927623 (+ 935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:06.927629 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:06.927640 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:22:06.927656 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:06.927705 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:22:06.927723 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:06.927848 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:06.928061 (+ 213us) server_negotiation.cc:299] Negotiation successful
0928 05:22:06.928252 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":322,"threads_started":1}
I20260928 05:22:06.931488 21036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ed4026e0eb14d39a39f26939fe51365: No bootstrap required, opened a new log
I20260928 05:22:06.932312 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 05:22:06.939320 21039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:06.937170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46387 (local address 127.0.0.1:33789)
0928 05:22:06.937346 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:06.937351 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:06.937696 (+ 345us) server_negotiation.cc:407] Connection header received
0928 05:22:06.937814 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:06.937819 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:06.937872 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:06.937979 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:06.938722 (+ 743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:06.938727 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:06.938732 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:22:06.938756 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:06.938797 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:22:06.938816 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:06.938932 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:06.938980 (+ 48us) server_negotiation.cc:299] Negotiation successful
0928 05:22:06.939033 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 05:22:06.940806 21036 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ed4026e0eb14d39a39f26939fe51365 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ed4026e0eb14d39a39f26939fe51365" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33789 } }
I20260928 05:22:06.941143 21036 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5ed4026e0eb14d39a39f26939fe51365 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:22:06.941212 21036 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ed4026e0eb14d39a39f26939fe51365 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ed4026e0eb14d39a39f26939fe51365, State: Initialized, Role: FOLLOWER
I20260928 05:22:06.941654 21036 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ed4026e0eb14d39a39f26939fe51365 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ed4026e0eb14d39a39f26939fe51365" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33789 } }
I20260928 05:22:06.941874 21036 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5ed4026e0eb14d39a39f26939fe51365 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:22:06.941965 21036 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ed4026e0eb14d39a39f26939fe51365 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:22:06.942081 21036 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ed4026e0eb14d39a39f26939fe51365 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:22:06.944021 21036 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ed4026e0eb14d39a39f26939fe51365 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ed4026e0eb14d39a39f26939fe51365" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33789 } }
I20260928 05:22:06.944424 21036 leader_election.cc:302] T 00000000000000000000000000000000 P 5ed4026e0eb14d39a39f26939fe51365 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ed4026e0eb14d39a39f26939fe51365; no voters:
I20260928 05:22:06.944929 21036 leader_election.cc:288] T 00000000000000000000000000000000 P 5ed4026e0eb14d39a39f26939fe51365 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:22:06.945086 21046 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ed4026e0eb14d39a39f26939fe51365 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:22:06.946096 21046 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ed4026e0eb14d39a39f26939fe51365 [term 1 LEADER]: Becoming Leader. State: Replica: 5ed4026e0eb14d39a39f26939fe51365, State: Running, Role: LEADER
I20260928 05:22:06.946570 21046 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ed4026e0eb14d39a39f26939fe51365 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ed4026e0eb14d39a39f26939fe51365" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33789 } }
I20260928 05:22:06.946862 21036 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ed4026e0eb14d39a39f26939fe51365 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:22:06.949317 21047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ed4026e0eb14d39a39f26939fe51365 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5ed4026e0eb14d39a39f26939fe51365" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ed4026e0eb14d39a39f26939fe51365" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33789 } } }
I20260928 05:22:06.949707 21047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ed4026e0eb14d39a39f26939fe51365 [sys.catalog]: This master's current role is: LEADER
I20260928 05:22:06.950124 21048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ed4026e0eb14d39a39f26939fe51365 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ed4026e0eb14d39a39f26939fe51365. Latest consensus state: current_term: 1 leader_uuid: "5ed4026e0eb14d39a39f26939fe51365" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ed4026e0eb14d39a39f26939fe51365" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33789 } } }
I20260928 05:22:06.950376 21048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ed4026e0eb14d39a39f26939fe51365 [sys.catalog]: This master's current role is: LEADER
I20260928 05:22:06.958369 21054 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:22:06.962199 21054 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:22:06.970203 21054 catalog_manager.cc:1347] Generated new cluster ID: 7e888d7fe8c74e13bbb36e2733647ae7
I20260928 05:22:06.970287 21054 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:22:06.997262 21054 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:22:06.997684 21054 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:22:07.009181 21054 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5ed4026e0eb14d39a39f26939fe51365: Generated new TSK 0
I20260928 05:22:07.009986 21054 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 05:22:07.024264 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 20969
W20260928 05:22:07.025574 21031 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:56835
W20260928 05:22:07.026216 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 20969: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 05:22:07.026351 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 20969
2026-09-28T05:22:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1725 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790572622576160-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:22:07 dist-test-slave-whsn krb5kdc[21067](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:22:07 dist-test-slave-whsn krb5kdc[21067](info): set up 2 sockets
Sep 28 05:22:07 dist-test-slave-whsn krb5kdc[21067](info): commencing operation
krb5kdc: starting...
W20260928 05:22:08.020771 388 mini_kdc.cc:121] Time spent starting KDC: real 0.963s user 0.001s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790572622576160-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790572622576160-388-0/krb5kdc/test-user.keytab.
Sep 28 05:22:08 dist-test-slave-whsn krb5kdc[21067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:22:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:22:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790572622576160-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790572622576160-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 05:22:08.230124 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790572622576160-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790572622576160-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790572622576160-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40867
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790572622576160-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40867
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:22:08.540009 21080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:22:08.540278 21080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:22:08.540414 21080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:22:08.548117 21080 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:22:08.548219 21080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:22:08.548269 21080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:22:08.548326 21080 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:22:08.548367 21080 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:22:08.559291 21080 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790572622576160-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790572622576160-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40867
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790572622576160-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790572622576160-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:22:08.560578 21080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:22:08.561882 21080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:22:08.573025 21085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:22:08.573079 21086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:22:08.573050 21088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:22:08.573802 21080 server_base.cc:1034] running on GCE node
I20260928 05:22:08.574527 21080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:22:08.575874 21080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:22:08.577072 21080 hybrid_clock.cc:648] HybridClock initialized: now 1790572928577014 us; error 45 us; skew 500 ppm
I20260928 05:22:08.577415 21080 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:22:08 dist-test-slave-whsn krb5kdc[21067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572928, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:22:08.580245 21080 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 05:22:08.581625 21080 webserver.cc:460] Webserver started at http://127.0.0.1:38665/ using document root <none> and password file <none>
I20260928 05:22:08.582230 21080 fs_manager.cc:359] Metadata directory not provided
I20260928 05:22:08.582324 21080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:22:08.582562 21080 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:22:08.585410 21080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790572622576160-388-0/minicluster-data/master-0/data/instance:
uuid: "4b9f196af29f4c2cb1f4048bdf332b99"
format_stamp: "Formatted at 2026-09-28 05:22:08 on dist-test-slave-whsn"
server_key: "3fda087f76bd4d1558b966f057b01530"
server_key_iv: "49a612ecea255d8a3805aac149596d52"
server_key_version: "encryptionkey@0"
I20260928 05:22:08.586082 21080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790572622576160-388-0/minicluster-data/master-0/wal/instance:
uuid: "4b9f196af29f4c2cb1f4048bdf332b99"
format_stamp: "Formatted at 2026-09-28 05:22:08 on dist-test-slave-whsn"
server_key: "3fda087f76bd4d1558b966f057b01530"
server_key_iv: "49a612ecea255d8a3805aac149596d52"
server_key_version: "encryptionkey@0"
I20260928 05:22:08.590803 21080 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 05:22:08.593880 21095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:22:08.595115 21080 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260928 05:22:08.595265 21080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790572622576160-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790572622576160-388-0/minicluster-data/master-0/wal
uuid: "4b9f196af29f4c2cb1f4048bdf332b99"
format_stamp: "Formatted at 2026-09-28 05:22:08 on dist-test-slave-whsn"
server_key: "3fda087f76bd4d1558b966f057b01530"
server_key_iv: "49a612ecea255d8a3805aac149596d52"
server_key_version: "encryptionkey@0"
I20260928 05:22:08.595460 21080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790572622576160-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790572622576160-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790572622576160-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:22:08.616643 21080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:22:08.620213 21080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:22:08.620450 21080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:22:08.641430 21080 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40867
I20260928 05:22:08.641465 21146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40867 every 8 connection(s)
I20260928 05:22:08.642701 21080 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:22:08.646193 21147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:22:08.648605 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 21080
I20260928 05:22:08.648757 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790572622576160-388-0/minicluster-data/master-0/wal/instance
I20260928 05:22:08.649186 388 external_mini_cluster.cc:1428] Setting key 15f022555c97673f72934cda7d9a3f1a
I20260928 05:22:08.656603 21147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b9f196af29f4c2cb1f4048bdf332b99: Bootstrap starting.
I20260928 05:22:08.659489 21147 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b9f196af29f4c2cb1f4048bdf332b99: Neither blocks nor log segments found. Creating new log.
I20260928 05:22:08.660455 21147 log.cc:824] T 00000000000000000000000000000000 P 4b9f196af29f4c2cb1f4048bdf332b99: Log is configured to *not* fsync() on all Append() calls
Sep 28 05:22:08 dist-test-slave-whsn krb5kdc[21067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 05:22:08.663161 21147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b9f196af29f4c2cb1f4048bdf332b99: No bootstrap required, opened a new log
I20260928 05:22:08.669723 21150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:08.650872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60241 (local address 127.0.0.1:40867)
0928 05:22:08.651558 (+ 686us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:08.651570 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:08.651608 (+ 38us) server_negotiation.cc:407] Connection header received
0928 05:22:08.653263 (+ 1655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:08.653284 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:08.653665 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:08.654014 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:08.655318 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:08.656618 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:08.657921 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:08.658568 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:08.658733 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:08.662826 (+ 4093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:08.662834 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:08.662848 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:22:08.662882 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:08.665877 (+ 2995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:08.666522 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:08.666531 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:08.666537 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:08.666713 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:08.667089 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:08.667093 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:08.667095 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:08.667613 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:08.667895 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:08.668288 (+ 393us) server_negotiation.cc:299] Negotiation successful
0928 05:22:08.668686 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":300,"threads_started":1}
I20260928 05:22:08.670941 21147 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b9f196af29f4c2cb1f4048bdf332b99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b9f196af29f4c2cb1f4048bdf332b99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40867 } }
I20260928 05:22:08.671269 21147 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b9f196af29f4c2cb1f4048bdf332b99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:22:08.671366 21147 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b9f196af29f4c2cb1f4048bdf332b99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b9f196af29f4c2cb1f4048bdf332b99, State: Initialized, Role: FOLLOWER
I20260928 05:22:08.671963 21147 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b9f196af29f4c2cb1f4048bdf332b99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b9f196af29f4c2cb1f4048bdf332b99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40867 } }
I20260928 05:22:08.672233 21147 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b9f196af29f4c2cb1f4048bdf332b99 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:22:08.672376 21147 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b9f196af29f4c2cb1f4048bdf332b99 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:22:08.672554 21147 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b9f196af29f4c2cb1f4048bdf332b99 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:22:08.673223 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 05:22:08.675573 21147 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b9f196af29f4c2cb1f4048bdf332b99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b9f196af29f4c2cb1f4048bdf332b99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40867 } }
I20260928 05:22:08.676117 21147 leader_election.cc:302] T 00000000000000000000000000000000 P 4b9f196af29f4c2cb1f4048bdf332b99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b9f196af29f4c2cb1f4048bdf332b99; no voters:
I20260928 05:22:08.676764 21147 leader_election.cc:288] T 00000000000000000000000000000000 P 4b9f196af29f4c2cb1f4048bdf332b99 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:22:08.676877 21154 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b9f196af29f4c2cb1f4048bdf332b99 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:22:08.677547 21154 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b9f196af29f4c2cb1f4048bdf332b99 [term 1 LEADER]: Becoming Leader. State: Replica: 4b9f196af29f4c2cb1f4048bdf332b99, State: Running, Role: LEADER
I20260928 05:22:08.678179 21154 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b9f196af29f4c2cb1f4048bdf332b99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b9f196af29f4c2cb1f4048bdf332b99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40867 } }
I20260928 05:22:08.679376 21147 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b9f196af29f4c2cb1f4048bdf332b99 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:22:08.681881 21157 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b9f196af29f4c2cb1f4048bdf332b99 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b9f196af29f4c2cb1f4048bdf332b99. Latest consensus state: current_term: 1 leader_uuid: "4b9f196af29f4c2cb1f4048bdf332b99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b9f196af29f4c2cb1f4048bdf332b99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40867 } } }
I20260928 05:22:08.682264 21157 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b9f196af29f4c2cb1f4048bdf332b99 [sys.catalog]: This master's current role is: LEADER
I20260928 05:22:08.682920 21158 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b9f196af29f4c2cb1f4048bdf332b99 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b9f196af29f4c2cb1f4048bdf332b99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b9f196af29f4c2cb1f4048bdf332b99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40867 } } }
I20260928 05:22:08.683612 21158 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b9f196af29f4c2cb1f4048bdf332b99 [sys.catalog]: This master's current role is: LEADER
I20260928 05:22:08.689865 21162 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:22:08.696231 21162 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:22:08.701193 21150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:08.678669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60385 (local address 127.0.0.1:40867)
0928 05:22:08.678834 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:08.678840 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:08.681506 (+ 2666us) server_negotiation.cc:407] Connection header received
0928 05:22:08.681847 (+ 341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:08.681853 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:08.681934 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:08.682008 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:08.685522 (+ 3514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:08.686624 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:08.688468 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:08.689175 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:08.689277 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:08.692397 (+ 3120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:08.692404 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:08.692410 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:22:08.692448 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:08.696070 (+ 3622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:08.696907 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:08.696912 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:08.696916 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:08.697084 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:08.699234 (+ 2150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:08.699239 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:08.699241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:08.699688 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:08.699912 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:08.700554 (+ 642us) server_negotiation.cc:299] Negotiation successful
0928 05:22:08.700773 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 05:22:08.708760 21162 catalog_manager.cc:1347] Generated new cluster ID: 40201c727c974ed3b7a01179e68871a8
I20260928 05:22:08.708871 21162 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:22:08.735124 21162 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:22:08.736929 21162 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:22:08.744030 21162 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b9f196af29f4c2cb1f4048bdf332b99: Generated new TSK 0
I20260928 05:22:08.745036 21162 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 05:22:08.759672 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 21080
I20260928 05:22:08.760612 21142 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:22:08.849565 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 21080
2026-09-28T05:22:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1823 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790572622576160-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:22:08 dist-test-slave-whsn krb5kdc[21179](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:22:08 dist-test-slave-whsn krb5kdc[21179](info): set up 2 sockets
Sep 28 05:22:08 dist-test-slave-whsn krb5kdc[21179](info): commencing operation
krb5kdc: starting...
W20260928 05:22:09.852754 388 mini_kdc.cc:121] Time spent starting KDC: real 0.972s user 0.004s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790572622576160-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790572622576160-388-0/krb5kdc/test-user.keytab.
Sep 28 05:22:09 dist-test-slave-whsn krb5kdc[21179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572929, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:22:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:22:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790572622576160-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790572622576160-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 05:22:10.061691 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790572622576160-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790572622576160-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790572622576160-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42413
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790572622576160-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42413
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:22:10.370978 21192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:22:10.371258 21192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:22:10.371402 21192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:22:10.378413 21192 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:22:10.378515 21192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:22:10.378597 21192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:22:10.378683 21192 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:22:10.378746 21192 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:22:10.388181 21192 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790572622576160-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790572622576160-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42413
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790572622576160-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42413
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790572622576160-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:22:10.389377 21192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:22:10.390605 21192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:22:10.401793 21200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:22:10.401880 21198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:22:10.401849 21197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:22:10.402661 21192 server_base.cc:1034] running on GCE node
I20260928 05:22:10.403371 21192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:22:10.404688 21192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:22:10.405905 21192 hybrid_clock.cc:648] HybridClock initialized: now 1790572930405848 us; error 40 us; skew 500 ppm
I20260928 05:22:10.406275 21192 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:22:10 dist-test-slave-whsn krb5kdc[21179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572930, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:22:10.409309 21192 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 05:22:10.410694 21192 webserver.cc:460] Webserver started at http://127.0.0.1:33803/ using document root <none> and password file <none>
I20260928 05:22:10.411309 21192 fs_manager.cc:359] Metadata directory not provided
I20260928 05:22:10.411420 21192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:22:10.411679 21192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:22:10.414448 21192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790572622576160-388-0/minicluster-data/master-0/data/instance:
uuid: "c2d882ff289043ec9885a2d7c1da96d2"
format_stamp: "Formatted at 2026-09-28 05:22:10 on dist-test-slave-whsn"
server_key: "58ce7fd9afd8418310608ad683bff3b4"
server_key_iv: "aac7756fbecc12c5c1e9a905042645be"
server_key_version: "encryptionkey@0"
I20260928 05:22:10.415136 21192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790572622576160-388-0/minicluster-data/master-0/wal/instance:
uuid: "c2d882ff289043ec9885a2d7c1da96d2"
format_stamp: "Formatted at 2026-09-28 05:22:10 on dist-test-slave-whsn"
server_key: "58ce7fd9afd8418310608ad683bff3b4"
server_key_iv: "aac7756fbecc12c5c1e9a905042645be"
server_key_version: "encryptionkey@0"
I20260928 05:22:10.419855 21192 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260928 05:22:10.422933 21207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:22:10.424079 21192 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260928 05:22:10.424222 21192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790572622576160-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790572622576160-388-0/minicluster-data/master-0/wal
uuid: "c2d882ff289043ec9885a2d7c1da96d2"
format_stamp: "Formatted at 2026-09-28 05:22:10 on dist-test-slave-whsn"
server_key: "58ce7fd9afd8418310608ad683bff3b4"
server_key_iv: "aac7756fbecc12c5c1e9a905042645be"
server_key_version: "encryptionkey@0"
I20260928 05:22:10.424386 21192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790572622576160-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790572622576160-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790572622576160-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:22:10.489261 21192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:22:10.493048 21192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:22:10.493305 21192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:22:10.512468 21192 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42413
I20260928 05:22:10.512491 21258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42413 every 8 connection(s)
I20260928 05:22:10.513805 21192 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:22:10.517347 21259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:22:10.521204 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 21192
I20260928 05:22:10.521381 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790572622576160-388-0/minicluster-data/master-0/wal/instance
I20260928 05:22:10.521872 388 external_mini_cluster.cc:1428] Setting key 72e455f385f26ba93a4aa0fca995d99e
I20260928 05:22:10.530049 21259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2d882ff289043ec9885a2d7c1da96d2: Bootstrap starting.
I20260928 05:22:10.533422 21259 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c2d882ff289043ec9885a2d7c1da96d2: Neither blocks nor log segments found. Creating new log.
I20260928 05:22:10.534511 21259 log.cc:824] T 00000000000000000000000000000000 P c2d882ff289043ec9885a2d7c1da96d2: Log is configured to *not* fsync() on all Append() calls
I20260928 05:22:10.535610 21262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:10.523579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50471 (local address 127.0.0.1:42413)
0928 05:22:10.524358 (+ 779us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:10.524368 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:10.524411 (+ 43us) server_negotiation.cc:407] Connection header received
0928 05:22:10.526141 (+ 1730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:10.526163 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:10.526579 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:10.527341 (+ 762us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:10.527844 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:10.529393 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:10.531153 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:10.531850 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:10.532003 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:10.533117 (+ 1114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:10.533126 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:10.533139 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:22:10.533711 (+ 572us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:10.533798 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:22:10.533827 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:10.533919 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:10.534147 (+ 228us) server_negotiation.cc:299] Negotiation successful
0928 05:22:10.534391 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":358,"threads_started":1}
I20260928 05:22:10.537544 21259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2d882ff289043ec9885a2d7c1da96d2: No bootstrap required, opened a new log
I20260928 05:22:10.539183 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 05:22:10.545857 21259 raft_consensus.cc:348] T 00000000000000000000000000000000 P c2d882ff289043ec9885a2d7c1da96d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2d882ff289043ec9885a2d7c1da96d2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42413 } }
I20260928 05:22:10.546300 21259 raft_consensus.cc:374] T 00000000000000000000000000000000 P c2d882ff289043ec9885a2d7c1da96d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:22:10.546406 21259 raft_consensus.cc:729] T 00000000000000000000000000000000 P c2d882ff289043ec9885a2d7c1da96d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2d882ff289043ec9885a2d7c1da96d2, State: Initialized, Role: FOLLOWER
I20260928 05:22:10.547042 21259 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2d882ff289043ec9885a2d7c1da96d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2d882ff289043ec9885a2d7c1da96d2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42413 } }
I20260928 05:22:10.547309 21259 raft_consensus.cc:388] T 00000000000000000000000000000000 P c2d882ff289043ec9885a2d7c1da96d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:22:10.547454 21259 raft_consensus.cc:482] T 00000000000000000000000000000000 P c2d882ff289043ec9885a2d7c1da96d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:22:10.547629 21259 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2d882ff289043ec9885a2d7c1da96d2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:22:10.549346 21262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:10.543071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42911 (local address 127.0.0.1:42413)
0928 05:22:10.543265 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:10.543271 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:10.543596 (+ 325us) server_negotiation.cc:407] Connection header received
0928 05:22:10.543753 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:10.543756 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:10.543815 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:10.543890 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:10.544492 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:10.545230 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:10.547005 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:10.547487 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:10.547606 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:10.548669 (+ 1063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:10.548673 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:10.548676 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:22:10.548692 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:10.548746 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:22:10.548760 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:10.548828 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:10.549000 (+ 172us) server_negotiation.cc:299] Negotiation successful
0928 05:22:10.549065 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 05:22:10.550578 21259 raft_consensus.cc:504] T 00000000000000000000000000000000 P c2d882ff289043ec9885a2d7c1da96d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2d882ff289043ec9885a2d7c1da96d2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42413 } }
I20260928 05:22:10.551028 21259 leader_election.cc:302] T 00000000000000000000000000000000 P c2d882ff289043ec9885a2d7c1da96d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c2d882ff289043ec9885a2d7c1da96d2; no voters:
I20260928 05:22:10.551528 21259 leader_election.cc:288] T 00000000000000000000000000000000 P c2d882ff289043ec9885a2d7c1da96d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:22:10.551605 21269 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c2d882ff289043ec9885a2d7c1da96d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:22:10.552789 21269 raft_consensus.cc:686] T 00000000000000000000000000000000 P c2d882ff289043ec9885a2d7c1da96d2 [term 1 LEADER]: Becoming Leader. State: Replica: c2d882ff289043ec9885a2d7c1da96d2, State: Running, Role: LEADER
I20260928 05:22:10.553303 21269 consensus_queue.cc:237] T 00000000000000000000000000000000 P c2d882ff289043ec9885a2d7c1da96d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2d882ff289043ec9885a2d7c1da96d2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42413 } }
I20260928 05:22:10.553584 21259 sys_catalog.cc:564] T 00000000000000000000000000000000 P c2d882ff289043ec9885a2d7c1da96d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:22:10.555974 21271 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2d882ff289043ec9885a2d7c1da96d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c2d882ff289043ec9885a2d7c1da96d2. Latest consensus state: current_term: 1 leader_uuid: "c2d882ff289043ec9885a2d7c1da96d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2d882ff289043ec9885a2d7c1da96d2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42413 } } }
I20260928 05:22:10.556313 21271 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2d882ff289043ec9885a2d7c1da96d2 [sys.catalog]: This master's current role is: LEADER
I20260928 05:22:10.556715 21270 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2d882ff289043ec9885a2d7c1da96d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c2d882ff289043ec9885a2d7c1da96d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2d882ff289043ec9885a2d7c1da96d2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42413 } } }
I20260928 05:22:10.557013 21270 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2d882ff289043ec9885a2d7c1da96d2 [sys.catalog]: This master's current role is: LEADER
I20260928 05:22:10.559526 21275 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:22:10.565122 21275 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:22:10.573231 21275 catalog_manager.cc:1347] Generated new cluster ID: d44eaba8b6ee406bae7ff62ad905c043
I20260928 05:22:10.573325 21275 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:22:10.584316 21275 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:22:10.585539 21275 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:22:10.593237 21275 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c2d882ff289043ec9885a2d7c1da96d2: Generated new TSK 0
I20260928 05:22:10.593935 21275 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 05:22:10.603163 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 21192
W20260928 05:22:10.604130 21254 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:50471
W20260928 05:22:10.604717 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 21192: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 05:22:10.604882 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 21192
2026-09-28T05:22:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1754 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790572622576160-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:22:10 dist-test-slave-whsn krb5kdc[21290](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:22:10 dist-test-slave-whsn krb5kdc[21290](info): set up 2 sockets
Sep 28 05:22:10 dist-test-slave-whsn krb5kdc[21290](info): commencing operation
krb5kdc: starting...
W20260928 05:22:11.612825 388 mini_kdc.cc:121] Time spent starting KDC: real 0.977s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790572622576160-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790572622576160-388-0/krb5kdc/test-user.keytab.
Sep 28 05:22:11 dist-test-slave-whsn krb5kdc[21290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572931, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:22:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:22:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790572622576160-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790572622576160-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 05:22:11.823721 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790572622576160-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790572622576160-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790572622576160-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45087
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790572622576160-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45087
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:22:12.129627 21303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:22:12.129900 21303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:22:12.130028 21303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:22:12.137102 21303 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:22:12.137200 21303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:22:12.137274 21303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:22:12.137353 21303 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:22:12.137418 21303 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:22:12.146688 21303 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790572622576160-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790572622576160-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45087
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790572622576160-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45087
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790572622576160-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:22:12.147852 21303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:22:12.149310 21303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:22:12.160223 21309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:22:12.160386 21308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:22:12.160418 21311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:22:12.161163 21303 server_base.cc:1034] running on GCE node
I20260928 05:22:12.161831 21303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:22:12.163098 21303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:22:12.164321 21303 hybrid_clock.cc:648] HybridClock initialized: now 1790572932164264 us; error 47 us; skew 500 ppm
I20260928 05:22:12.164649 21303 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:22:12 dist-test-slave-whsn krb5kdc[21290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572932, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:22:12.175366 21303 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 05:22:12.176712 21303 webserver.cc:460] Webserver started at http://127.0.0.1:39375/ using document root <none> and password file <none>
I20260928 05:22:12.177390 21303 fs_manager.cc:359] Metadata directory not provided
I20260928 05:22:12.177502 21303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:22:12.177778 21303 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:22:12.180575 21303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790572622576160-388-0/minicluster-data/master-0/data/instance:
uuid: "e28dcc54e1c74e1cba017f44dad528ad"
format_stamp: "Formatted at 2026-09-28 05:22:12 on dist-test-slave-whsn"
server_key: "045a0030d8061230f7b74ea1fcddefc9"
server_key_iv: "a4a391f3d3f82036347db06b1396b5a8"
server_key_version: "encryptionkey@0"
I20260928 05:22:12.181309 21303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790572622576160-388-0/minicluster-data/master-0/wal/instance:
uuid: "e28dcc54e1c74e1cba017f44dad528ad"
format_stamp: "Formatted at 2026-09-28 05:22:12 on dist-test-slave-whsn"
server_key: "045a0030d8061230f7b74ea1fcddefc9"
server_key_iv: "a4a391f3d3f82036347db06b1396b5a8"
server_key_version: "encryptionkey@0"
I20260928 05:22:12.185956 21303 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 05:22:12.189105 21318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:22:12.190268 21303 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260928 05:22:12.190420 21303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790572622576160-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790572622576160-388-0/minicluster-data/master-0/wal
uuid: "e28dcc54e1c74e1cba017f44dad528ad"
format_stamp: "Formatted at 2026-09-28 05:22:12 on dist-test-slave-whsn"
server_key: "045a0030d8061230f7b74ea1fcddefc9"
server_key_iv: "a4a391f3d3f82036347db06b1396b5a8"
server_key_version: "encryptionkey@0"
I20260928 05:22:12.190593 21303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790572622576160-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790572622576160-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790572622576160-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:22:12.195792 21303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:22:12.199229 21303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:22:12.199476 21303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:22:12.215591 21303 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45087
I20260928 05:22:12.215623 21369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45087 every 8 connection(s)
I20260928 05:22:12.217085 21303 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:22:12.220431 21370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:22:12.222921 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 21303
I20260928 05:22:12.223088 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790572622576160-388-0/minicluster-data/master-0/wal/instance
I20260928 05:22:12.223580 388 external_mini_cluster.cc:1428] Setting key 2e702a1af22c381add9d648bd6f7c5e3
I20260928 05:22:12.230556 21373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:12.225226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45373 (local address 127.0.0.1:45087)
0928 05:22:12.225964 (+ 738us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:12.225976 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:12.226014 (+ 38us) server_negotiation.cc:407] Connection header received
0928 05:22:12.227589 (+ 1575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:12.227609 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:12.227906 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:12.228213 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:12.229162 (+ 949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:12.229171 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:12.229183 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:22:12.229199 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:12.229255 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:22:12.229276 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:12.229437 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:12.229526 (+ 89us) server_negotiation.cc:299] Negotiation successful
0928 05:22:12.229701 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":327,"threads_started":1}
I20260928 05:22:12.231550 21370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e28dcc54e1c74e1cba017f44dad528ad: Bootstrap starting.
I20260928 05:22:12.233830 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 05:22:12.234604 21370 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e28dcc54e1c74e1cba017f44dad528ad: Neither blocks nor log segments found. Creating new log.
I20260928 05:22:12.235721 21370 log.cc:824] T 00000000000000000000000000000000 P e28dcc54e1c74e1cba017f44dad528ad: Log is configured to *not* fsync() on all Append() calls
I20260928 05:22:12.238696 21370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e28dcc54e1c74e1cba017f44dad528ad: No bootstrap required, opened a new log
I20260928 05:22:12.240782 21373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:12.238690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34899 (local address 127.0.0.1:45087)
0928 05:22:12.238818 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:12.238823 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:12.239121 (+ 298us) server_negotiation.cc:407] Connection header received
0928 05:22:12.239280 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:12.239283 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:12.239323 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:12.239394 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:12.240208 (+ 814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:12.240212 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:12.240215 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:22:12.240227 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:12.240261 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:22:12.240274 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:12.240365 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:12.240468 (+ 103us) server_negotiation.cc:299] Negotiation successful
0928 05:22:12.240510 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 05:22:12.246140 21370 raft_consensus.cc:348] T 00000000000000000000000000000000 P e28dcc54e1c74e1cba017f44dad528ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e28dcc54e1c74e1cba017f44dad528ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45087 } }
I20260928 05:22:12.246469 21370 raft_consensus.cc:374] T 00000000000000000000000000000000 P e28dcc54e1c74e1cba017f44dad528ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:22:12.246577 21370 raft_consensus.cc:729] T 00000000000000000000000000000000 P e28dcc54e1c74e1cba017f44dad528ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e28dcc54e1c74e1cba017f44dad528ad, State: Initialized, Role: FOLLOWER
I20260928 05:22:12.247146 21370 consensus_queue.cc:260] T 00000000000000000000000000000000 P e28dcc54e1c74e1cba017f44dad528ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e28dcc54e1c74e1cba017f44dad528ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45087 } }
I20260928 05:22:12.247349 21370 raft_consensus.cc:388] T 00000000000000000000000000000000 P e28dcc54e1c74e1cba017f44dad528ad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:22:12.247444 21370 raft_consensus.cc:482] T 00000000000000000000000000000000 P e28dcc54e1c74e1cba017f44dad528ad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:22:12.247565 21370 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e28dcc54e1c74e1cba017f44dad528ad [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:22:12.249596 21370 raft_consensus.cc:504] T 00000000000000000000000000000000 P e28dcc54e1c74e1cba017f44dad528ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e28dcc54e1c74e1cba017f44dad528ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45087 } }
I20260928 05:22:12.250000 21370 leader_election.cc:302] T 00000000000000000000000000000000 P e28dcc54e1c74e1cba017f44dad528ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e28dcc54e1c74e1cba017f44dad528ad; no voters:
I20260928 05:22:12.250432 21370 leader_election.cc:288] T 00000000000000000000000000000000 P e28dcc54e1c74e1cba017f44dad528ad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:22:12.250568 21380 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e28dcc54e1c74e1cba017f44dad528ad [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:22:12.251164 21380 raft_consensus.cc:686] T 00000000000000000000000000000000 P e28dcc54e1c74e1cba017f44dad528ad [term 1 LEADER]: Becoming Leader. State: Replica: e28dcc54e1c74e1cba017f44dad528ad, State: Running, Role: LEADER
I20260928 05:22:12.251650 21380 consensus_queue.cc:237] T 00000000000000000000000000000000 P e28dcc54e1c74e1cba017f44dad528ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e28dcc54e1c74e1cba017f44dad528ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45087 } }
I20260928 05:22:12.252245 21370 sys_catalog.cc:564] T 00000000000000000000000000000000 P e28dcc54e1c74e1cba017f44dad528ad [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:22:12.254122 21381 sys_catalog.cc:455] T 00000000000000000000000000000000 P e28dcc54e1c74e1cba017f44dad528ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e28dcc54e1c74e1cba017f44dad528ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e28dcc54e1c74e1cba017f44dad528ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45087 } } }
I20260928 05:22:12.254510 21381 sys_catalog.cc:458] T 00000000000000000000000000000000 P e28dcc54e1c74e1cba017f44dad528ad [sys.catalog]: This master's current role is: LEADER
I20260928 05:22:12.254907 21382 sys_catalog.cc:455] T 00000000000000000000000000000000 P e28dcc54e1c74e1cba017f44dad528ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader e28dcc54e1c74e1cba017f44dad528ad. Latest consensus state: current_term: 1 leader_uuid: "e28dcc54e1c74e1cba017f44dad528ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e28dcc54e1c74e1cba017f44dad528ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45087 } } }
I20260928 05:22:12.255189 21382 sys_catalog.cc:458] T 00000000000000000000000000000000 P e28dcc54e1c74e1cba017f44dad528ad [sys.catalog]: This master's current role is: LEADER
I20260928 05:22:12.255861 21388 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:22:12.260633 21388 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:22:12.268945 21388 catalog_manager.cc:1347] Generated new cluster ID: a9bf9f4d88d04f048c81d3f85f676ffb
I20260928 05:22:12.269050 21388 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:22:12.308109 21388 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:22:12.308511 21388 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:22:12.315390 21388 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e28dcc54e1c74e1cba017f44dad528ad: Generated new TSK 0
I20260928 05:22:12.316128 21388 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 05:22:12.328222 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 21303
W20260928 05:22:12.329267 21365 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:45373
W20260928 05:22:12.329870 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 21303: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 05:22:12.329988 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 21303
2026-09-28T05:22:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1726 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790572622576160-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:22:12 dist-test-slave-whsn krb5kdc[21401](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:22:12 dist-test-slave-whsn krb5kdc[21401](info): set up 2 sockets
Sep 28 05:22:12 dist-test-slave-whsn krb5kdc[21401](info): commencing operation
krb5kdc: starting...
W20260928 05:22:13.332999 388 mini_kdc.cc:121] Time spent starting KDC: real 0.970s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790572622576160-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790572622576160-388-0/krb5kdc/test-user.keytab.
Sep 28 05:22:13 dist-test-slave-whsn krb5kdc[21401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572933, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:22:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:22:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790572622576160-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790572622576160-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 05:22:13.540225 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790572622576160-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790572622576160-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790572622576160-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41431
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790572622576160-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41431
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:22:13.840945 21414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:22:13.841224 21414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:22:13.841361 21414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:22:13.848272 21414 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:22:13.848384 21414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:22:13.848460 21414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:22:13.848536 21414 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:22:13.848595 21414 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:22:13.857935 21414 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790572622576160-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790572622576160-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41431
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790572622576160-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790572622576160-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:22:13.859098 21414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:22:13.860308 21414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:22:13.870977 21419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:22:13.870940 21420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:22:13.871013 21422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:22:13.872008 21414 server_base.cc:1034] running on GCE node
I20260928 05:22:13.872752 21414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:22:13.874068 21414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:22:13.875272 21414 hybrid_clock.cc:648] HybridClock initialized: now 1790572933875227 us; error 36 us; skew 500 ppm
I20260928 05:22:13.875581 21414 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:22:13 dist-test-slave-whsn krb5kdc[21401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572933, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:22:13.878427 21414 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 05:22:13.879776 21414 webserver.cc:460] Webserver started at http://127.0.0.1:43587/ using document root <none> and password file <none>
I20260928 05:22:13.880388 21414 fs_manager.cc:359] Metadata directory not provided
I20260928 05:22:13.880486 21414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:22:13.880759 21414 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:22:13.883509 21414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790572622576160-388-0/minicluster-data/master-0/data/instance:
uuid: "4872ccdac64f47c0b473c12f8fcbb34a"
format_stamp: "Formatted at 2026-09-28 05:22:13 on dist-test-slave-whsn"
server_key: "f21fcd60416848d252442a88029167a8"
server_key_iv: "75f9a9ca2e6479ded42afec81edba544"
server_key_version: "encryptionkey@0"
I20260928 05:22:13.884202 21414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790572622576160-388-0/minicluster-data/master-0/wal/instance:
uuid: "4872ccdac64f47c0b473c12f8fcbb34a"
format_stamp: "Formatted at 2026-09-28 05:22:13 on dist-test-slave-whsn"
server_key: "f21fcd60416848d252442a88029167a8"
server_key_iv: "75f9a9ca2e6479ded42afec81edba544"
server_key_version: "encryptionkey@0"
I20260928 05:22:13.888819 21414 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260928 05:22:13.891930 21429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:22:13.893185 21414 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 05:22:13.893333 21414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790572622576160-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790572622576160-388-0/minicluster-data/master-0/wal
uuid: "4872ccdac64f47c0b473c12f8fcbb34a"
format_stamp: "Formatted at 2026-09-28 05:22:13 on dist-test-slave-whsn"
server_key: "f21fcd60416848d252442a88029167a8"
server_key_iv: "75f9a9ca2e6479ded42afec81edba544"
server_key_version: "encryptionkey@0"
I20260928 05:22:13.893493 21414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790572622576160-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790572622576160-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790572622576160-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:22:13.905169 21414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:22:13.908726 21414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:22:13.908991 21414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:22:13.925267 21414 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41431
I20260928 05:22:13.925338 21480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41431 every 8 connection(s)
I20260928 05:22:13.926506 21414 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:22:13.929912 21481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:22:13.936388 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 21414
I20260928 05:22:13.936573 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790572622576160-388-0/minicluster-data/master-0/wal/instance
I20260928 05:22:13.937094 388 external_mini_cluster.cc:1428] Setting key d835e74a6b4262f8786e00a228bb4d82
I20260928 05:22:13.939910 21481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4872ccdac64f47c0b473c12f8fcbb34a: Bootstrap starting.
I20260928 05:22:13.943075 21481 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4872ccdac64f47c0b473c12f8fcbb34a: Neither blocks nor log segments found. Creating new log.
I20260928 05:22:13.944109 21481 log.cc:824] T 00000000000000000000000000000000 P 4872ccdac64f47c0b473c12f8fcbb34a: Log is configured to *not* fsync() on all Append() calls
I20260928 05:22:13.947101 21481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4872ccdac64f47c0b473c12f8fcbb34a: No bootstrap required, opened a new log
Sep 28 05:22:13 dist-test-slave-whsn krb5kdc[21401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572933, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 05:22:13.956595 21481 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4872ccdac64f47c0b473c12f8fcbb34a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4872ccdac64f47c0b473c12f8fcbb34a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41431 } }
I20260928 05:22:13.957050 21481 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4872ccdac64f47c0b473c12f8fcbb34a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:22:13.957172 21481 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4872ccdac64f47c0b473c12f8fcbb34a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4872ccdac64f47c0b473c12f8fcbb34a, State: Initialized, Role: FOLLOWER
I20260928 05:22:13.957798 21481 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4872ccdac64f47c0b473c12f8fcbb34a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4872ccdac64f47c0b473c12f8fcbb34a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41431 } }
I20260928 05:22:13.958086 21481 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4872ccdac64f47c0b473c12f8fcbb34a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:22:13.958237 21481 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4872ccdac64f47c0b473c12f8fcbb34a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:22:13.958434 21481 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4872ccdac64f47c0b473c12f8fcbb34a [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:22:13.960532 21484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:13.938692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.143:42395 (local address 127.0.0.1:41431)
0928 05:22:13.939364 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:13.939375 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:13.939414 (+ 39us) server_negotiation.cc:407] Connection header received
0928 05:22:13.941094 (+ 1680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:13.941119 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:13.941495 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:13.941794 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:13.943518 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:13.945083 (+ 1565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:13.946892 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:13.947566 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:13.947689 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:13.952064 (+ 4375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:13.952073 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:13.952091 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:22:13.952137 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:13.956007 (+ 3870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:13.956744 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:13.956752 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:13.956760 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:13.956971 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:13.957489 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:13.957493 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:13.957496 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:13.958193 (+ 697us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:13.958507 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:13.958926 (+ 419us) server_negotiation.cc:299] Negotiation successful
0928 05:22:13.959261 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":319,"threads_started":1}
I20260928 05:22:13.961611 21481 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4872ccdac64f47c0b473c12f8fcbb34a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4872ccdac64f47c0b473c12f8fcbb34a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41431 } }
I20260928 05:22:13.962124 21481 leader_election.cc:302] T 00000000000000000000000000000000 P 4872ccdac64f47c0b473c12f8fcbb34a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4872ccdac64f47c0b473c12f8fcbb34a; no voters:
I20260928 05:22:13.962662 21481 leader_election.cc:288] T 00000000000000000000000000000000 P 4872ccdac64f47c0b473c12f8fcbb34a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:22:13.962798 21486 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4872ccdac64f47c0b473c12f8fcbb34a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:22:13.963816 21486 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4872ccdac64f47c0b473c12f8fcbb34a [term 1 LEADER]: Becoming Leader. State: Replica: 4872ccdac64f47c0b473c12f8fcbb34a, State: Running, Role: LEADER
I20260928 05:22:13.964375 21486 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4872ccdac64f47c0b473c12f8fcbb34a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4872ccdac64f47c0b473c12f8fcbb34a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41431 } }
I20260928 05:22:13.964581 21481 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4872ccdac64f47c0b473c12f8fcbb34a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:22:13.964843 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 05:22:13.967039 21488 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4872ccdac64f47c0b473c12f8fcbb34a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4872ccdac64f47c0b473c12f8fcbb34a. Latest consensus state: current_term: 1 leader_uuid: "4872ccdac64f47c0b473c12f8fcbb34a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4872ccdac64f47c0b473c12f8fcbb34a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41431 } } }
I20260928 05:22:13.967039 21487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4872ccdac64f47c0b473c12f8fcbb34a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4872ccdac64f47c0b473c12f8fcbb34a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4872ccdac64f47c0b473c12f8fcbb34a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41431 } } }
I20260928 05:22:13.967460 21487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4872ccdac64f47c0b473c12f8fcbb34a [sys.catalog]: This master's current role is: LEADER
I20260928 05:22:13.967460 21488 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4872ccdac64f47c0b473c12f8fcbb34a [sys.catalog]: This master's current role is: LEADER
I20260928 05:22:13.973644 21492 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:22:13.979447 21492 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:22:14.001287 21492 catalog_manager.cc:1347] Generated new cluster ID: 779bc0e428b5482e9b46ba8caf742a47
I20260928 05:22:14.001386 21484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:13.976317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.143:54271 (local address 127.0.0.1:41431)
0928 05:22:13.976489 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:13.976496 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:13.983975 (+ 7479us) server_negotiation.cc:407] Connection header received
0928 05:22:13.984267 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:13.984273 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:13.984364 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:13.984444 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:13.986475 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:13.987710 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:13.989674 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:13.990388 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:13.990480 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:13.993927 (+ 3447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:13.993935 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:13.993940 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:22:13.993982 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:13.997618 (+ 3636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:13.998661 (+ 1043us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:13.998667 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:13.998671 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:13.998832 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:13.999363 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:13.999368 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:13.999372 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:13.999839 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:14.000116 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:14.000777 (+ 661us) server_negotiation.cc:299] Negotiation successful
0928 05:22:14.000986 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 05:22:14.001396 21492 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:22:14.014971 21492 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:22:14.016700 21492 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:22:14.025794 21492 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4872ccdac64f47c0b473c12f8fcbb34a: Generated new TSK 0
I20260928 05:22:14.026693 21492 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 05:22:14.031281 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 21414
I20260928 05:22:14.033082 21476 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:22:14.120586 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 21414
W20260928 05:22:14.134250 21399 connection.cc:535] client connection to 127.0.0.1:41431 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41431): Connection reset by peer (error 104)
2026-09-28T05:22:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1789 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790572622576160-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:22:14 dist-test-slave-whsn krb5kdc[21513](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:22:14 dist-test-slave-whsn krb5kdc[21513](info): set up 2 sockets
Sep 28 05:22:14 dist-test-slave-whsn krb5kdc[21513](info): commencing operation
krb5kdc: starting...
W20260928 05:22:15.120939 388 mini_kdc.cc:121] Time spent starting KDC: real 0.969s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790572622576160-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790572622576160-388-0/krb5kdc/test-user.keytab.
Sep 28 05:22:15 dist-test-slave-whsn krb5kdc[21513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572935, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:22:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:22:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790572622576160-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790572622576160-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 05:22:15.339874 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790572622576160-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790572622576160-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790572622576160-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37867
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790572622576160-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37867
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:22:15.643016 21526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:22:15.643290 21526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:22:15.643417 21526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:22:15.650460 21526 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:22:15.650559 21526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:22:15.650629 21526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:22:15.650697 21526 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:22:15.650758 21526 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:22:15.660151 21526 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790572622576160-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790572622576160-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37867
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790572622576160-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790572622576160-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:22:15.661412 21526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:22:15.662644 21526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:22:15.673736 21531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:22:15.673768 21532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:22:15.673823 21534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:22:15.674252 21526 server_base.cc:1034] running on GCE node
I20260928 05:22:15.675040 21526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:22:15.676380 21526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:22:15.677567 21526 hybrid_clock.cc:648] HybridClock initialized: now 1790572935677506 us; error 54 us; skew 500 ppm
I20260928 05:22:15.677858 21526 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:22:15 dist-test-slave-whsn krb5kdc[21513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572935, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:22:15.680673 21526 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 05:22:15.681931 21526 webserver.cc:460] Webserver started at http://127.0.0.1:37303/ using document root <none> and password file <none>
I20260928 05:22:15.682505 21526 fs_manager.cc:359] Metadata directory not provided
I20260928 05:22:15.682595 21526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:22:15.682828 21526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:22:15.685560 21526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790572622576160-388-0/minicluster-data/master-0/data/instance:
uuid: "74fb86453f8241a8938f956328ff46ab"
format_stamp: "Formatted at 2026-09-28 05:22:15 on dist-test-slave-whsn"
server_key: "bedddc63e043cb01aacc1456c4980e24"
server_key_iv: "393107106bc148296870f902844cad11"
server_key_version: "encryptionkey@0"
I20260928 05:22:15.686198 21526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790572622576160-388-0/minicluster-data/master-0/wal/instance:
uuid: "74fb86453f8241a8938f956328ff46ab"
format_stamp: "Formatted at 2026-09-28 05:22:15 on dist-test-slave-whsn"
server_key: "bedddc63e043cb01aacc1456c4980e24"
server_key_iv: "393107106bc148296870f902844cad11"
server_key_version: "encryptionkey@0"
I20260928 05:22:15.690754 21526 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 05:22:15.693753 21541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:22:15.694877 21526 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260928 05:22:15.695004 21526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790572622576160-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790572622576160-388-0/minicluster-data/master-0/wal
uuid: "74fb86453f8241a8938f956328ff46ab"
format_stamp: "Formatted at 2026-09-28 05:22:15 on dist-test-slave-whsn"
server_key: "bedddc63e043cb01aacc1456c4980e24"
server_key_iv: "393107106bc148296870f902844cad11"
server_key_version: "encryptionkey@0"
I20260928 05:22:15.695152 21526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790572622576160-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790572622576160-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790572622576160-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:22:15.707808 21526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:22:15.711553 21526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:22:15.711760 21526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:22:15.727558 21526 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37867
I20260928 05:22:15.727576 21592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37867 every 8 connection(s)
I20260928 05:22:15.728770 21526 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:22:15.732136 21593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:22:15.737951 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 21526
I20260928 05:22:15.738111 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790572622576160-388-0/minicluster-data/master-0/wal/instance
I20260928 05:22:15.738504 388 external_mini_cluster.cc:1428] Setting key 94f7f649ca69e12b80e63e7ceeb2240e
I20260928 05:22:15.742050 21593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74fb86453f8241a8938f956328ff46ab: Bootstrap starting.
I20260928 05:22:15.745296 21593 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 74fb86453f8241a8938f956328ff46ab: Neither blocks nor log segments found. Creating new log.
I20260928 05:22:15.746469 21593 log.cc:824] T 00000000000000000000000000000000 P 74fb86453f8241a8938f956328ff46ab: Log is configured to *not* fsync() on all Append() calls
I20260928 05:22:15.749518 21593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74fb86453f8241a8938f956328ff46ab: No bootstrap required, opened a new log
I20260928 05:22:15.751157 21596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:15.740028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.143:35647 (local address 127.0.0.1:37867)
0928 05:22:15.740698 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:15.740708 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:15.740746 (+ 38us) server_negotiation.cc:407] Connection header received
0928 05:22:15.742391 (+ 1645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:15.742412 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:15.742791 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:15.743088 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:15.743651 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:15.744890 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:15.746763 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:15.747761 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:15.747880 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:15.749063 (+ 1183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:15.749070 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:15.749083 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:22:15.749590 (+ 507us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:15.749667 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:22:15.749690 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:15.749804 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:15.750072 (+ 268us) server_negotiation.cc:299] Negotiation successful
0928 05:22:15.750256 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":296,"threads_started":1}
I20260928 05:22:15.754520 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 05:22:15.757087 21593 raft_consensus.cc:348] T 00000000000000000000000000000000 P 74fb86453f8241a8938f956328ff46ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74fb86453f8241a8938f956328ff46ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37867 } }
I20260928 05:22:15.757418 21593 raft_consensus.cc:374] T 00000000000000000000000000000000 P 74fb86453f8241a8938f956328ff46ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:22:15.757512 21593 raft_consensus.cc:729] T 00000000000000000000000000000000 P 74fb86453f8241a8938f956328ff46ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74fb86453f8241a8938f956328ff46ab, State: Initialized, Role: FOLLOWER
I20260928 05:22:15.758049 21593 consensus_queue.cc:260] T 00000000000000000000000000000000 P 74fb86453f8241a8938f956328ff46ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74fb86453f8241a8938f956328ff46ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37867 } }
I20260928 05:22:15.758271 21593 raft_consensus.cc:388] T 00000000000000000000000000000000 P 74fb86453f8241a8938f956328ff46ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:22:15.758385 21593 raft_consensus.cc:482] T 00000000000000000000000000000000 P 74fb86453f8241a8938f956328ff46ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:22:15.758599 21593 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 74fb86453f8241a8938f956328ff46ab [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:22:15.761359 21593 raft_consensus.cc:504] T 00000000000000000000000000000000 P 74fb86453f8241a8938f956328ff46ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74fb86453f8241a8938f956328ff46ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37867 } }
I20260928 05:22:15.761885 21593 leader_election.cc:302] T 00000000000000000000000000000000 P 74fb86453f8241a8938f956328ff46ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 74fb86453f8241a8938f956328ff46ab; no voters:
I20260928 05:22:15.762473 21593 leader_election.cc:288] T 00000000000000000000000000000000 P 74fb86453f8241a8938f956328ff46ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:22:15.762552 21603 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 74fb86453f8241a8938f956328ff46ab [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:22:15.763523 21603 raft_consensus.cc:686] T 00000000000000000000000000000000 P 74fb86453f8241a8938f956328ff46ab [term 1 LEADER]: Becoming Leader. State: Replica: 74fb86453f8241a8938f956328ff46ab, State: Running, Role: LEADER
I20260928 05:22:15.764137 21603 consensus_queue.cc:237] T 00000000000000000000000000000000 P 74fb86453f8241a8938f956328ff46ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74fb86453f8241a8938f956328ff46ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37867 } }
I20260928 05:22:15.764452 21593 sys_catalog.cc:564] T 00000000000000000000000000000000 P 74fb86453f8241a8938f956328ff46ab [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:22:15.765779 21596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:15.758709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.143:34807 (local address 127.0.0.1:37867)
0928 05:22:15.758864 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:15.758869 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:15.759344 (+ 475us) server_negotiation.cc:407] Connection header received
0928 05:22:15.759501 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:15.759505 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:15.759565 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:15.759648 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:15.760218 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:15.760965 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:15.762716 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:15.763442 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:15.763575 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:15.764930 (+ 1355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:15.764936 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:15.764942 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:22:15.764963 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:15.765028 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:22:15.765050 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:15.765167 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:15.765371 (+ 204us) server_negotiation.cc:299] Negotiation successful
0928 05:22:15.765423 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 05:22:15.774487 21604 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74fb86453f8241a8938f956328ff46ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "74fb86453f8241a8938f956328ff46ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74fb86453f8241a8938f956328ff46ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37867 } } }
I20260928 05:22:15.774423 21605 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74fb86453f8241a8938f956328ff46ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 74fb86453f8241a8938f956328ff46ab. Latest consensus state: current_term: 1 leader_uuid: "74fb86453f8241a8938f956328ff46ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74fb86453f8241a8938f956328ff46ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37867 } } }
I20260928 05:22:15.774955 21605 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74fb86453f8241a8938f956328ff46ab [sys.catalog]: This master's current role is: LEADER
I20260928 05:22:15.774953 21604 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74fb86453f8241a8938f956328ff46ab [sys.catalog]: This master's current role is: LEADER
I20260928 05:22:15.775928 21617 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:22:15.780138 21617 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:22:15.787526 21617 catalog_manager.cc:1347] Generated new cluster ID: 8ab90f23477848929b149b0c2a900463
I20260928 05:22:15.787616 21617 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:22:15.796664 21617 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:22:15.797874 21617 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:22:15.804141 21617 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 74fb86453f8241a8938f956328ff46ab: Generated new TSK 0
I20260928 05:22:15.804841 21617 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 05:22:15.815054 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 21526
W20260928 05:22:15.816124 21588 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.143:35647
W20260928 05:22:15.816817 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 21526: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 05:22:15.816980 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 21526
W20260928 05:22:15.830564 21511 connection.cc:535] client connection to 127.0.0.1:37867 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37867): Connection reset by peer (error 104)
2026-09-28T05:22:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1696 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790572622576160-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:22:15 dist-test-slave-whsn krb5kdc[21625](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:22:15 dist-test-slave-whsn krb5kdc[21625](info): set up 2 sockets
Sep 28 05:22:15 dist-test-slave-whsn krb5kdc[21625](info): commencing operation
krb5kdc: starting...
W20260928 05:22:16.816830 388 mini_kdc.cc:121] Time spent starting KDC: real 0.968s user 0.001s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790572622576160-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790572622576160-388-0/krb5kdc/test-user.keytab.
Sep 28 05:22:16 dist-test-slave-whsn krb5kdc[21625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572936, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:22:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:22:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790572622576160-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790572622576160-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 05:22:17.035898 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790572622576160-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790572622576160-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790572622576160-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43447
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790572622576160-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43447
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:22:17.343029 21638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:22:17.343318 21638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:22:17.343453 21638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:22:17.350498 21638 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:22:17.350620 21638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:22:17.350694 21638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:22:17.350776 21638 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:22:17.350834 21638 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:22:17.360328 21638 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790572622576160-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790572622576160-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43447
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790572622576160-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790572622576160-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:22:17.361557 21638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:22:17.362784 21638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:22:17.373546 21643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:22:17.373631 21644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:22:17.373634 21646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:22:17.374634 21638 server_base.cc:1034] running on GCE node
I20260928 05:22:17.375330 21638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:22:17.376610 21638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:22:17.377810 21638 hybrid_clock.cc:648] HybridClock initialized: now 1790572937377747 us; error 47 us; skew 500 ppm
I20260928 05:22:17.378140 21638 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:22:17 dist-test-slave-whsn krb5kdc[21625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572937, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:22:17.385435 21638 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 05:22:17.386821 21638 webserver.cc:460] Webserver started at http://127.0.0.1:40141/ using document root <none> and password file <none>
I20260928 05:22:17.387483 21638 fs_manager.cc:359] Metadata directory not provided
I20260928 05:22:17.387604 21638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:22:17.387887 21638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:22:17.390817 21638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790572622576160-388-0/minicluster-data/master-0/data/instance:
uuid: "458d63721e044a519da7b90397bb84f0"
format_stamp: "Formatted at 2026-09-28 05:22:17 on dist-test-slave-whsn"
server_key: "e3bad6a6591c7762f2252b63ac193d14"
server_key_iv: "7faece10d6c8a4ad6e3a782c7207a515"
server_key_version: "encryptionkey@0"
I20260928 05:22:17.391547 21638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790572622576160-388-0/minicluster-data/master-0/wal/instance:
uuid: "458d63721e044a519da7b90397bb84f0"
format_stamp: "Formatted at 2026-09-28 05:22:17 on dist-test-slave-whsn"
server_key: "e3bad6a6591c7762f2252b63ac193d14"
server_key_iv: "7faece10d6c8a4ad6e3a782c7207a515"
server_key_version: "encryptionkey@0"
I20260928 05:22:17.396440 21638 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 05:22:17.399704 21653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:22:17.400954 21638 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:22:17.401124 21638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790572622576160-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790572622576160-388-0/minicluster-data/master-0/wal
uuid: "458d63721e044a519da7b90397bb84f0"
format_stamp: "Formatted at 2026-09-28 05:22:17 on dist-test-slave-whsn"
server_key: "e3bad6a6591c7762f2252b63ac193d14"
server_key_iv: "7faece10d6c8a4ad6e3a782c7207a515"
server_key_version: "encryptionkey@0"
I20260928 05:22:17.401289 21638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790572622576160-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790572622576160-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790572622576160-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:22:17.406507 21638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:22:17.410249 21638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:22:17.410511 21638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:22:17.427074 21638 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43447
I20260928 05:22:17.427095 21704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43447 every 8 connection(s)
I20260928 05:22:17.428340 21638 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:22:17.431674 21705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:22:17.434948 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 21638
I20260928 05:22:17.435125 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790572622576160-388-0/minicluster-data/master-0/wal/instance
I20260928 05:22:17.435616 388 external_mini_cluster.cc:1428] Setting key c990fc8c73365d48d80f01498633173e
I20260928 05:22:17.442972 21708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:17.437331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.143:41797 (local address 127.0.0.1:43447)
0928 05:22:17.438040 (+ 709us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:17.438056 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:17.438098 (+ 42us) server_negotiation.cc:407] Connection header received
0928 05:22:17.439786 (+ 1688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:17.439807 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:17.440138 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:17.440489 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:17.441493 (+ 1004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:17.441499 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:17.441511 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:22:17.441526 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:17.441589 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:22:17.441610 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:17.441815 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:17.441951 (+ 136us) server_negotiation.cc:299] Negotiation successful
0928 05:22:17.442121 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":499,"thread_start_us":282,"threads_started":1}
I20260928 05:22:17.443142 21705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 458d63721e044a519da7b90397bb84f0: Bootstrap starting.
I20260928 05:22:17.446111 21705 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 458d63721e044a519da7b90397bb84f0: Neither blocks nor log segments found. Creating new log.
I20260928 05:22:17.446305 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 05:22:17.447062 21705 log.cc:824] T 00000000000000000000000000000000 P 458d63721e044a519da7b90397bb84f0: Log is configured to *not* fsync() on all Append() calls
I20260928 05:22:17.449721 21705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 458d63721e044a519da7b90397bb84f0: No bootstrap required, opened a new log
I20260928 05:22:17.452953 21708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:17.450693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.143:56187 (local address 127.0.0.1:43447)
0928 05:22:17.450851 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:17.450856 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:17.451245 (+ 389us) server_negotiation.cc:407] Connection header received
0928 05:22:17.451371 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:17.451377 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:17.451435 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:17.451539 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:17.452333 (+ 794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:17.452342 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:17.452347 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:22:17.452364 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:17.452406 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:22:17.452424 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:17.452542 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:17.452612 (+ 70us) server_negotiation.cc:299] Negotiation successful
0928 05:22:17.452662 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 05:22:17.457232 21705 raft_consensus.cc:348] T 00000000000000000000000000000000 P 458d63721e044a519da7b90397bb84f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "458d63721e044a519da7b90397bb84f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43447 } }
I20260928 05:22:17.457541 21705 raft_consensus.cc:374] T 00000000000000000000000000000000 P 458d63721e044a519da7b90397bb84f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:22:17.457633 21705 raft_consensus.cc:729] T 00000000000000000000000000000000 P 458d63721e044a519da7b90397bb84f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 458d63721e044a519da7b90397bb84f0, State: Initialized, Role: FOLLOWER
I20260928 05:22:17.458163 21705 consensus_queue.cc:260] T 00000000000000000000000000000000 P 458d63721e044a519da7b90397bb84f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "458d63721e044a519da7b90397bb84f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43447 } }
I20260928 05:22:17.458376 21705 raft_consensus.cc:388] T 00000000000000000000000000000000 P 458d63721e044a519da7b90397bb84f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:22:17.458498 21705 raft_consensus.cc:482] T 00000000000000000000000000000000 P 458d63721e044a519da7b90397bb84f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:22:17.458639 21705 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 458d63721e044a519da7b90397bb84f0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:22:17.460590 21705 raft_consensus.cc:504] T 00000000000000000000000000000000 P 458d63721e044a519da7b90397bb84f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "458d63721e044a519da7b90397bb84f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43447 } }
I20260928 05:22:17.461093 21705 leader_election.cc:302] T 00000000000000000000000000000000 P 458d63721e044a519da7b90397bb84f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 458d63721e044a519da7b90397bb84f0; no voters:
I20260928 05:22:17.461608 21705 leader_election.cc:288] T 00000000000000000000000000000000 P 458d63721e044a519da7b90397bb84f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:22:17.461719 21715 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 458d63721e044a519da7b90397bb84f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:22:17.462297 21715 raft_consensus.cc:686] T 00000000000000000000000000000000 P 458d63721e044a519da7b90397bb84f0 [term 1 LEADER]: Becoming Leader. State: Replica: 458d63721e044a519da7b90397bb84f0, State: Running, Role: LEADER
I20260928 05:22:17.462749 21715 consensus_queue.cc:237] T 00000000000000000000000000000000 P 458d63721e044a519da7b90397bb84f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "458d63721e044a519da7b90397bb84f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43447 } }
I20260928 05:22:17.463332 21705 sys_catalog.cc:564] T 00000000000000000000000000000000 P 458d63721e044a519da7b90397bb84f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:22:17.465258 21716 sys_catalog.cc:455] T 00000000000000000000000000000000 P 458d63721e044a519da7b90397bb84f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "458d63721e044a519da7b90397bb84f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "458d63721e044a519da7b90397bb84f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43447 } } }
I20260928 05:22:17.465652 21716 sys_catalog.cc:458] T 00000000000000000000000000000000 P 458d63721e044a519da7b90397bb84f0 [sys.catalog]: This master's current role is: LEADER
I20260928 05:22:17.465588 21717 sys_catalog.cc:455] T 00000000000000000000000000000000 P 458d63721e044a519da7b90397bb84f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 458d63721e044a519da7b90397bb84f0. Latest consensus state: current_term: 1 leader_uuid: "458d63721e044a519da7b90397bb84f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "458d63721e044a519da7b90397bb84f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43447 } } }
I20260928 05:22:17.465869 21717 sys_catalog.cc:458] T 00000000000000000000000000000000 P 458d63721e044a519da7b90397bb84f0 [sys.catalog]: This master's current role is: LEADER
I20260928 05:22:17.467144 21724 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:22:17.471145 21724 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:22:17.478108 21724 catalog_manager.cc:1347] Generated new cluster ID: 253e3d2cf8174c9499f9f2c13c300013
I20260928 05:22:17.478205 21724 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:22:17.496965 21724 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:22:17.497335 21724 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:22:17.508068 21724 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 458d63721e044a519da7b90397bb84f0: Generated new TSK 0
I20260928 05:22:17.508745 21724 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 05:22:17.518786 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 21638
W20260928 05:22:17.519703 21700 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.143:41797
W20260928 05:22:17.520397 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 21638: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 05:22:17.520516 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 21638
2026-09-28T05:22:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1703 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790572622576160-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:22:17 dist-test-slave-whsn krb5kdc[21736](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:22:17 dist-test-slave-whsn krb5kdc[21736](info): set up 2 sockets
Sep 28 05:22:17 dist-test-slave-whsn krb5kdc[21736](info): commencing operation
krb5kdc: starting...
W20260928 05:22:18.524911 388 mini_kdc.cc:121] Time spent starting KDC: real 0.973s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790572622576160-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790572622576160-388-0/krb5kdc/test-user.keytab.
Sep 28 05:22:18 dist-test-slave-whsn krb5kdc[21736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572938, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:22:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:22:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790572622576160-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790572622576160-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 05:22:18.735332 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790572622576160-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790572622576160-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790572622576160-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38057
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790572622576160-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38057
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:22:19.047326 21749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:22:19.047585 21749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:22:19.047712 21749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:22:19.054702 21749 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:22:19.054804 21749 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:22:19.054874 21749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:22:19.054953 21749 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:22:19.055009 21749 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:22:19.064428 21749 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790572622576160-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790572622576160-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38057
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790572622576160-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38057
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790572622576160-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:22:19.065656 21749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:22:19.066927 21749 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:22:19.078145 21757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:22:19.078164 21754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:22:19.078195 21755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:22:19.079124 21749 server_base.cc:1034] running on GCE node
I20260928 05:22:19.079829 21749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:22:19.081146 21749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:22:19.082340 21749 hybrid_clock.cc:648] HybridClock initialized: now 1790572939082295 us; error 38 us; skew 500 ppm
I20260928 05:22:19.082654 21749 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:22:19 dist-test-slave-whsn krb5kdc[21736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572939, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:22:19.085625 21749 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 05:22:19.086943 21749 webserver.cc:460] Webserver started at http://127.0.0.1:41097/ using document root <none> and password file <none>
I20260928 05:22:19.087572 21749 fs_manager.cc:359] Metadata directory not provided
I20260928 05:22:19.087694 21749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:22:19.087968 21749 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:22:19.090824 21749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790572622576160-388-0/minicluster-data/master-0/data/instance:
uuid: "1656dbb554e14889a27701d0b03942f7"
format_stamp: "Formatted at 2026-09-28 05:22:19 on dist-test-slave-whsn"
server_key: "9b2e5a0ca67913fba6336935796da5ba"
server_key_iv: "4fa745bdd3be03d0a32a597830a6ec25"
server_key_version: "encryptionkey@0"
I20260928 05:22:19.091530 21749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790572622576160-388-0/minicluster-data/master-0/wal/instance:
uuid: "1656dbb554e14889a27701d0b03942f7"
format_stamp: "Formatted at 2026-09-28 05:22:19 on dist-test-slave-whsn"
server_key: "9b2e5a0ca67913fba6336935796da5ba"
server_key_iv: "4fa745bdd3be03d0a32a597830a6ec25"
server_key_version: "encryptionkey@0"
I20260928 05:22:19.096285 21749 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260928 05:22:19.099443 21764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:22:19.100637 21749 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260928 05:22:19.100780 21749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790572622576160-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790572622576160-388-0/minicluster-data/master-0/wal
uuid: "1656dbb554e14889a27701d0b03942f7"
format_stamp: "Formatted at 2026-09-28 05:22:19 on dist-test-slave-whsn"
server_key: "9b2e5a0ca67913fba6336935796da5ba"
server_key_iv: "4fa745bdd3be03d0a32a597830a6ec25"
server_key_version: "encryptionkey@0"
I20260928 05:22:19.100975 21749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790572622576160-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790572622576160-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790572622576160-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:22:19.118752 21749 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:22:19.122960 21749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:22:19.123211 21749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:22:19.138978 21749 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38057
I20260928 05:22:19.139008 21815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38057 every 8 connection(s)
I20260928 05:22:19.140210 21749 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:22:19.143740 21816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:22:19.144371 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 21749
I20260928 05:22:19.144528 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790572622576160-388-0/minicluster-data/master-0/wal/instance
I20260928 05:22:19.144966 388 external_mini_cluster.cc:1428] Setting key b10470268c5339d18c19431f53478f90
I20260928 05:22:19.157059 21816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1656dbb554e14889a27701d0b03942f7: Bootstrap starting.
Sep 28 05:22:19 dist-test-slave-whsn krb5kdc[21736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572938, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 05:22:19.159840 21816 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1656dbb554e14889a27701d0b03942f7: Neither blocks nor log segments found. Creating new log.
I20260928 05:22:19.161067 21816 log.cc:824] T 00000000000000000000000000000000 P 1656dbb554e14889a27701d0b03942f7: Log is configured to *not* fsync() on all Append() calls
I20260928 05:22:19.163750 21816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1656dbb554e14889a27701d0b03942f7: No bootstrap required, opened a new log
I20260928 05:22:19.167053 21819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:19.146614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.143:44675 (local address 127.0.0.1:38057)
0928 05:22:19.147333 (+ 719us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:19.147347 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:19.147394 (+ 47us) server_negotiation.cc:407] Connection header received
0928 05:22:19.149785 (+ 2391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:19.149812 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:19.150346 (+ 534us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:19.150780 (+ 434us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:19.151998 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:19.153947 (+ 1949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:19.155168 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:19.156142 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:19.156324 (+ 182us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:19.159935 (+ 3611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:19.159946 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:19.159963 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:22:19.160007 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:19.163197 (+ 3190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:19.163887 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:19.163895 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:19.163903 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:19.164061 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:19.164486 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:19.164490 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:19.164492 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:19.165029 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:19.165345 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:19.165814 (+ 469us) server_negotiation.cc:299] Negotiation successful
0928 05:22:19.166099 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":283,"threads_started":1}
I20260928 05:22:19.170670 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 05:22:19.171819 21816 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1656dbb554e14889a27701d0b03942f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1656dbb554e14889a27701d0b03942f7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38057 } }
I20260928 05:22:19.172138 21816 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1656dbb554e14889a27701d0b03942f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:22:19.172221 21816 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1656dbb554e14889a27701d0b03942f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1656dbb554e14889a27701d0b03942f7, State: Initialized, Role: FOLLOWER
I20260928 05:22:19.172744 21816 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1656dbb554e14889a27701d0b03942f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1656dbb554e14889a27701d0b03942f7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38057 } }
I20260928 05:22:19.172986 21816 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1656dbb554e14889a27701d0b03942f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:22:19.173115 21816 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1656dbb554e14889a27701d0b03942f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:22:19.173280 21816 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1656dbb554e14889a27701d0b03942f7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:22:19.175662 21816 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1656dbb554e14889a27701d0b03942f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1656dbb554e14889a27701d0b03942f7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38057 } }
I20260928 05:22:19.176097 21816 leader_election.cc:302] T 00000000000000000000000000000000 P 1656dbb554e14889a27701d0b03942f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1656dbb554e14889a27701d0b03942f7; no voters:
I20260928 05:22:19.176628 21816 leader_election.cc:288] T 00000000000000000000000000000000 P 1656dbb554e14889a27701d0b03942f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:22:19.176736 21826 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1656dbb554e14889a27701d0b03942f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:22:19.177325 21826 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1656dbb554e14889a27701d0b03942f7 [term 1 LEADER]: Becoming Leader. State: Replica: 1656dbb554e14889a27701d0b03942f7, State: Running, Role: LEADER
I20260928 05:22:19.177878 21826 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1656dbb554e14889a27701d0b03942f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1656dbb554e14889a27701d0b03942f7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38057 } }
I20260928 05:22:19.179481 21816 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1656dbb554e14889a27701d0b03942f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:22:19.183135 21827 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1656dbb554e14889a27701d0b03942f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1656dbb554e14889a27701d0b03942f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1656dbb554e14889a27701d0b03942f7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38057 } } }
I20260928 05:22:19.183445 21827 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1656dbb554e14889a27701d0b03942f7 [sys.catalog]: This master's current role is: LEADER
I20260928 05:22:19.184204 21833 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:22:19.184122 21828 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1656dbb554e14889a27701d0b03942f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1656dbb554e14889a27701d0b03942f7. Latest consensus state: current_term: 1 leader_uuid: "1656dbb554e14889a27701d0b03942f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1656dbb554e14889a27701d0b03942f7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38057 } } }
I20260928 05:22:19.184394 21828 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1656dbb554e14889a27701d0b03942f7 [sys.catalog]: This master's current role is: LEADER
I20260928 05:22:19.192726 21819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:19.174695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.143:35561 (local address 127.0.0.1:38057)
0928 05:22:19.174829 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:19.174835 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:19.175211 (+ 376us) server_negotiation.cc:407] Connection header received
0928 05:22:19.175363 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:19.175367 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:19.175421 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:19.175495 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:19.176893 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:19.178114 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:19.179791 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:19.180413 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:19.180513 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:19.185193 (+ 4680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:19.185201 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:19.185207 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:22:19.185246 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:19.189018 (+ 3772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:19.189988 (+ 970us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:19.189995 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:19.189998 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:19.190147 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:19.190705 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:19.190710 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:19.190714 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:19.191151 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:19.191389 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:19.192090 (+ 701us) server_negotiation.cc:299] Negotiation successful
0928 05:22:19.192304 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 05:22:19.195999 21833 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:22:19.206106 21833 catalog_manager.cc:1347] Generated new cluster ID: 6fac8cfb8ae941dcb53eceaf5d564de1
I20260928 05:22:19.206207 21833 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:22:19.218319 21833 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:22:19.219460 21833 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:22:19.225257 21833 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1656dbb554e14889a27701d0b03942f7: Generated new TSK 0
I20260928 05:22:19.225937 21833 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 05:22:19.237198 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 21749
I20260928 05:22:19.238345 21811 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:22:19.325228 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 21749
W20260928 05:22:19.338573 21734 connection.cc:535] client connection to 127.0.0.1:38057 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38057): Connection reset by peer (error 104)
2026-09-28T05:22:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1804 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790572622576160-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:22:19 dist-test-slave-whsn krb5kdc[21848](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:22:19 dist-test-slave-whsn krb5kdc[21848](info): set up 2 sockets
Sep 28 05:22:19 dist-test-slave-whsn krb5kdc[21848](info): commencing operation
krb5kdc: starting...
W20260928 05:22:20.332809 388 mini_kdc.cc:121] Time spent starting KDC: real 0.976s user 0.001s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790572622576160-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790572622576160-388-0/krb5kdc/test-user.keytab.
Sep 28 05:22:20 dist-test-slave-whsn krb5kdc[21848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572940, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:22:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:22:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790572622576160-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790572622576160-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 05:22:20.546216 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790572622576160-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790572622576160-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790572622576160-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41919
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790572622576160-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41919
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:22:20.851400 21861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:22:20.851684 21861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:22:20.851811 21861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:22:20.859014 21861 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:22:20.859122 21861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:22:20.859192 21861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:22:20.859272 21861 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:22:20.859326 21861 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:22:20.869141 21861 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790572622576160-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790572622576160-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41919
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790572622576160-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41919
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790572622576160-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:22:20.870411 21861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:22:20.871666 21861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:22:20.882946 21869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:22:20.883013 21866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:22:20.882990 21867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:22:20.883811 21861 server_base.cc:1034] running on GCE node
I20260928 05:22:20.884583 21861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:22:20.885987 21861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:22:20.887240 21861 hybrid_clock.cc:648] HybridClock initialized: now 1790572940887189 us; error 43 us; skew 500 ppm
I20260928 05:22:20.887562 21861 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:22:20 dist-test-slave-whsn krb5kdc[21848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572940, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:22:20.892836 21861 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 05:22:20.894286 21861 webserver.cc:460] Webserver started at http://127.0.0.1:46847/ using document root <none> and password file <none>
I20260928 05:22:20.894968 21861 fs_manager.cc:359] Metadata directory not provided
I20260928 05:22:20.895083 21861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:22:20.895366 21861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:22:20.898327 21861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790572622576160-388-0/minicluster-data/master-0/data/instance:
uuid: "735ed29c2ff449fa971c9a66e1bc0b0c"
format_stamp: "Formatted at 2026-09-28 05:22:20 on dist-test-slave-whsn"
server_key: "42238cb969463200521a89e8753e7fb1"
server_key_iv: "2caa9d495fc641afec5ff72fa246ec72"
server_key_version: "encryptionkey@0"
I20260928 05:22:20.899034 21861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790572622576160-388-0/minicluster-data/master-0/wal/instance:
uuid: "735ed29c2ff449fa971c9a66e1bc0b0c"
format_stamp: "Formatted at 2026-09-28 05:22:20 on dist-test-slave-whsn"
server_key: "42238cb969463200521a89e8753e7fb1"
server_key_iv: "2caa9d495fc641afec5ff72fa246ec72"
server_key_version: "encryptionkey@0"
I20260928 05:22:20.903968 21861 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260928 05:22:20.907294 21876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:22:20.908645 21861 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 05:22:20.908804 21861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790572622576160-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790572622576160-388-0/minicluster-data/master-0/wal
uuid: "735ed29c2ff449fa971c9a66e1bc0b0c"
format_stamp: "Formatted at 2026-09-28 05:22:20 on dist-test-slave-whsn"
server_key: "42238cb969463200521a89e8753e7fb1"
server_key_iv: "2caa9d495fc641afec5ff72fa246ec72"
server_key_version: "encryptionkey@0"
I20260928 05:22:20.909021 21861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790572622576160-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790572622576160-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790572622576160-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:22:20.927140 21861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:22:20.931058 21861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:22:20.931293 21861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:22:20.946628 21861 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41919
I20260928 05:22:20.946672 21927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41919 every 8 connection(s)
I20260928 05:22:20.947836 21861 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:22:20.951215 21928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:22:20.954563 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 21861
I20260928 05:22:20.954715 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790572622576160-388-0/minicluster-data/master-0/wal/instance
I20260928 05:22:20.955090 388 external_mini_cluster.cc:1428] Setting key 6809a693436c182a7830a3c25f14559b
I20260928 05:22:20.962792 21928 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 735ed29c2ff449fa971c9a66e1bc0b0c: Bootstrap starting.
I20260928 05:22:20.965852 21928 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 735ed29c2ff449fa971c9a66e1bc0b0c: Neither blocks nor log segments found. Creating new log.
I20260928 05:22:20.966921 21928 log.cc:824] T 00000000000000000000000000000000 P 735ed29c2ff449fa971c9a66e1bc0b0c: Log is configured to *not* fsync() on all Append() calls
I20260928 05:22:20.967483 21931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:20.956637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.143:36947 (local address 127.0.0.1:41919)
0928 05:22:20.957349 (+ 712us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:20.957360 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:20.957399 (+ 39us) server_negotiation.cc:407] Connection header received
0928 05:22:20.958998 (+ 1599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:20.959019 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:20.959413 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:20.959745 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:20.960308 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:20.961662 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:20.963339 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:20.964008 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:20.964177 (+ 169us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:20.965349 (+ 1172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:20.965357 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:20.965370 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:22:20.965871 (+ 501us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:20.965952 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:22:20.965982 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:20.966127 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:20.966383 (+ 256us) server_negotiation.cc:299] Negotiation successful
0928 05:22:20.966526 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":327,"threads_started":1}
I20260928 05:22:20.969827 21928 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 735ed29c2ff449fa971c9a66e1bc0b0c: No bootstrap required, opened a new log
I20260928 05:22:20.971141 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 05:22:20.977922 21928 raft_consensus.cc:348] T 00000000000000000000000000000000 P 735ed29c2ff449fa971c9a66e1bc0b0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "735ed29c2ff449fa971c9a66e1bc0b0c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41919 } }
I20260928 05:22:20.978417 21928 raft_consensus.cc:374] T 00000000000000000000000000000000 P 735ed29c2ff449fa971c9a66e1bc0b0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:22:20.978528 21928 raft_consensus.cc:729] T 00000000000000000000000000000000 P 735ed29c2ff449fa971c9a66e1bc0b0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 735ed29c2ff449fa971c9a66e1bc0b0c, State: Initialized, Role: FOLLOWER
I20260928 05:22:20.979156 21928 consensus_queue.cc:260] T 00000000000000000000000000000000 P 735ed29c2ff449fa971c9a66e1bc0b0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "735ed29c2ff449fa971c9a66e1bc0b0c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41919 } }
I20260928 05:22:20.979391 21928 raft_consensus.cc:388] T 00000000000000000000000000000000 P 735ed29c2ff449fa971c9a66e1bc0b0c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:22:20.979534 21928 raft_consensus.cc:482] T 00000000000000000000000000000000 P 735ed29c2ff449fa971c9a66e1bc0b0c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:22:20.979727 21928 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 735ed29c2ff449fa971c9a66e1bc0b0c [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:22:20.981407 21931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:20.975116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.143:50939 (local address 127.0.0.1:41919)
0928 05:22:20.975295 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:20.975301 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:20.975625 (+ 324us) server_negotiation.cc:407] Connection header received
0928 05:22:20.975794 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:20.975800 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:20.975861 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:20.975958 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:20.976491 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:20.977279 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:20.979016 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:20.979467 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:20.979571 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:20.980675 (+ 1104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:20.980683 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:20.980686 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:22:20.980702 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:20.980740 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:22:20.980756 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:20.980844 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:20.981061 (+ 217us) server_negotiation.cc:299] Negotiation successful
0928 05:22:20.981103 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 05:22:20.982506 21928 raft_consensus.cc:504] T 00000000000000000000000000000000 P 735ed29c2ff449fa971c9a66e1bc0b0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "735ed29c2ff449fa971c9a66e1bc0b0c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41919 } }
I20260928 05:22:20.983008 21928 leader_election.cc:302] T 00000000000000000000000000000000 P 735ed29c2ff449fa971c9a66e1bc0b0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 735ed29c2ff449fa971c9a66e1bc0b0c; no voters:
I20260928 05:22:20.983532 21928 leader_election.cc:288] T 00000000000000000000000000000000 P 735ed29c2ff449fa971c9a66e1bc0b0c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:22:20.983640 21938 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 735ed29c2ff449fa971c9a66e1bc0b0c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:22:20.984743 21938 raft_consensus.cc:686] T 00000000000000000000000000000000 P 735ed29c2ff449fa971c9a66e1bc0b0c [term 1 LEADER]: Becoming Leader. State: Replica: 735ed29c2ff449fa971c9a66e1bc0b0c, State: Running, Role: LEADER
I20260928 05:22:20.985280 21938 consensus_queue.cc:237] T 00000000000000000000000000000000 P 735ed29c2ff449fa971c9a66e1bc0b0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "735ed29c2ff449fa971c9a66e1bc0b0c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41919 } }
I20260928 05:22:20.985715 21928 sys_catalog.cc:564] T 00000000000000000000000000000000 P 735ed29c2ff449fa971c9a66e1bc0b0c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:22:20.987803 21939 sys_catalog.cc:455] T 00000000000000000000000000000000 P 735ed29c2ff449fa971c9a66e1bc0b0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "735ed29c2ff449fa971c9a66e1bc0b0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "735ed29c2ff449fa971c9a66e1bc0b0c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41919 } } }
I20260928 05:22:20.987789 21940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 735ed29c2ff449fa971c9a66e1bc0b0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 735ed29c2ff449fa971c9a66e1bc0b0c. Latest consensus state: current_term: 1 leader_uuid: "735ed29c2ff449fa971c9a66e1bc0b0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "735ed29c2ff449fa971c9a66e1bc0b0c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41919 } } }
I20260928 05:22:20.988191 21940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 735ed29c2ff449fa971c9a66e1bc0b0c [sys.catalog]: This master's current role is: LEADER
I20260928 05:22:20.988191 21939 sys_catalog.cc:458] T 00000000000000000000000000000000 P 735ed29c2ff449fa971c9a66e1bc0b0c [sys.catalog]: This master's current role is: LEADER
I20260928 05:22:20.990724 21943 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:22:20.995947 21943 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:22:21.008826 21943 catalog_manager.cc:1347] Generated new cluster ID: 3f526a7bd45649dd9802d8584ad63da5
I20260928 05:22:21.008993 21943 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:22:21.032054 21943 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:22:21.033895 21943 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:22:21.042596 21943 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 735ed29c2ff449fa971c9a66e1bc0b0c: Generated new TSK 0
I20260928 05:22:21.043355 21943 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 05:22:21.048903 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 21861
W20260928 05:22:21.050236 21923 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.143:36947
W20260928 05:22:21.050892 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 21861: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 05:22:21.051034 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 21861
W20260928 05:22:21.064349 21846 connection.cc:535] client connection to 127.0.0.1:41919 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41919): Connection reset by peer (error 104)
2026-09-28T05:22:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1725 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790572622576160-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:22:21 dist-test-slave-whsn krb5kdc[21959](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:22:21 dist-test-slave-whsn krb5kdc[21959](info): set up 2 sockets
Sep 28 05:22:21 dist-test-slave-whsn krb5kdc[21959](info): commencing operation
krb5kdc: starting...
W20260928 05:22:22.052884 388 mini_kdc.cc:121] Time spent starting KDC: real 0.970s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790572622576160-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790572622576160-388-0/krb5kdc/test-user.keytab.
Sep 28 05:22:22 dist-test-slave-whsn krb5kdc[21959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572942, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:22:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:22:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790572622576160-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790572622576160-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 05:22:22.271858 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790572622576160-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790572622576160-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790572622576160-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38005
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790572622576160-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38005
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:22:22.578059 21972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:22:22.578338 21972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:22:22.578474 21972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:22:22.585387 21972 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:22:22.585493 21972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:22:22.585566 21972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:22:22.585646 21972 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:22:22.585703 21972 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:22:22.594892 21972 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790572622576160-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790572622576160-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38005
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790572622576160-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38005
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790572622576160-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:22:22.596051 21972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:22:22.597295 21972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:22:22.607887 21977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:22:22.608209 21980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:22:22.608278 21978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:22:22.609295 21972 server_base.cc:1034] running on GCE node
I20260928 05:22:22.609979 21972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:22:22.611334 21972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:22:22.612545 21972 hybrid_clock.cc:648] HybridClock initialized: now 1790572942612488 us; error 47 us; skew 500 ppm
I20260928 05:22:22.612888 21972 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:22:22 dist-test-slave-whsn krb5kdc[21959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572942, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:22:22.619859 21972 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 05:22:22.621269 21972 webserver.cc:460] Webserver started at http://127.0.0.1:38035/ using document root <none> and password file <none>
I20260928 05:22:22.621888 21972 fs_manager.cc:359] Metadata directory not provided
I20260928 05:22:22.621982 21972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:22:22.622224 21972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:22:22.625005 21972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790572622576160-388-0/minicluster-data/master-0/data/instance:
uuid: "73efa44fd0214b3ab20f67187524323f"
format_stamp: "Formatted at 2026-09-28 05:22:22 on dist-test-slave-whsn"
server_key: "b22501ccd7f278c5d141aef4860b113f"
server_key_iv: "f4d6e5eeca7e1a40f7de8d049c40173b"
server_key_version: "encryptionkey@0"
I20260928 05:22:22.625681 21972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790572622576160-388-0/minicluster-data/master-0/wal/instance:
uuid: "73efa44fd0214b3ab20f67187524323f"
format_stamp: "Formatted at 2026-09-28 05:22:22 on dist-test-slave-whsn"
server_key: "b22501ccd7f278c5d141aef4860b113f"
server_key_iv: "f4d6e5eeca7e1a40f7de8d049c40173b"
server_key_version: "encryptionkey@0"
I20260928 05:22:22.630422 21972 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 05:22:22.633630 21987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:22:22.634871 21972 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 05:22:22.634999 21972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790572622576160-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790572622576160-388-0/minicluster-data/master-0/wal
uuid: "73efa44fd0214b3ab20f67187524323f"
format_stamp: "Formatted at 2026-09-28 05:22:22 on dist-test-slave-whsn"
server_key: "b22501ccd7f278c5d141aef4860b113f"
server_key_iv: "f4d6e5eeca7e1a40f7de8d049c40173b"
server_key_version: "encryptionkey@0"
I20260928 05:22:22.635165 21972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790572622576160-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790572622576160-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790572622576160-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:22:22.640277 21972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:22:22.644017 21972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:22:22.644248 21972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:22:22.660254 21972 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38005
I20260928 05:22:22.660320 22038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38005 every 8 connection(s)
I20260928 05:22:22.661535 21972 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:22:22.664916 22039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:22:22.669771 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 21972
I20260928 05:22:22.669941 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790572622576160-388-0/minicluster-data/master-0/wal/instance
I20260928 05:22:22.670429 388 external_mini_cluster.cc:1428] Setting key 980f2be6fdd852effb6b84deac213b15
I20260928 05:22:22.675588 22039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73efa44fd0214b3ab20f67187524323f: Bootstrap starting.
I20260928 05:22:22.677806 22042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:22.672048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.143:54417 (local address 127.0.0.1:38005)
0928 05:22:22.672803 (+ 755us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:22.672814 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:22.672875 (+ 61us) server_negotiation.cc:407] Connection header received
0928 05:22:22.674669 (+ 1794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:22.674698 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:22.675056 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:22.675357 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:22.676321 (+ 964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:22.676327 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:22.676339 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:22:22.676350 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:22.676395 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:22:22.676411 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:22.676559 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:22.676727 (+ 168us) server_negotiation.cc:299] Negotiation successful
0928 05:22:22.676956 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":335,"threads_started":1}
I20260928 05:22:22.678802 22039 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 73efa44fd0214b3ab20f67187524323f: Neither blocks nor log segments found. Creating new log.
I20260928 05:22:22.679752 22039 log.cc:824] T 00000000000000000000000000000000 P 73efa44fd0214b3ab20f67187524323f: Log is configured to *not* fsync() on all Append() calls
I20260928 05:22:22.681242 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 05:22:22.682860 22039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73efa44fd0214b3ab20f67187524323f: No bootstrap required, opened a new log
I20260928 05:22:22.688333 22042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:22.686111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.143:43735 (local address 127.0.0.1:38005)
0928 05:22:22.686284 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:22.686290 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:22.686613 (+ 323us) server_negotiation.cc:407] Connection header received
0928 05:22:22.686775 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:22.686781 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:22.686838 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:22.686910 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:22.687735 (+ 825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:22.687739 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:22.687742 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:22:22.687755 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:22.687797 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:22:22.687811 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:22.687899 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:22.688006 (+ 107us) server_negotiation.cc:299] Negotiation successful
0928 05:22:22.688054 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 05:22:22.691589 22039 raft_consensus.cc:348] T 00000000000000000000000000000000 P 73efa44fd0214b3ab20f67187524323f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73efa44fd0214b3ab20f67187524323f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38005 } }
I20260928 05:22:22.691913 22039 raft_consensus.cc:374] T 00000000000000000000000000000000 P 73efa44fd0214b3ab20f67187524323f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:22:22.691982 22039 raft_consensus.cc:729] T 00000000000000000000000000000000 P 73efa44fd0214b3ab20f67187524323f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73efa44fd0214b3ab20f67187524323f, State: Initialized, Role: FOLLOWER
I20260928 05:22:22.692433 22039 consensus_queue.cc:260] T 00000000000000000000000000000000 P 73efa44fd0214b3ab20f67187524323f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73efa44fd0214b3ab20f67187524323f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38005 } }
I20260928 05:22:22.692628 22039 raft_consensus.cc:388] T 00000000000000000000000000000000 P 73efa44fd0214b3ab20f67187524323f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:22:22.692719 22039 raft_consensus.cc:482] T 00000000000000000000000000000000 P 73efa44fd0214b3ab20f67187524323f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:22:22.692845 22039 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 73efa44fd0214b3ab20f67187524323f [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:22:22.694860 22039 raft_consensus.cc:504] T 00000000000000000000000000000000 P 73efa44fd0214b3ab20f67187524323f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73efa44fd0214b3ab20f67187524323f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38005 } }
I20260928 05:22:22.695344 22039 leader_election.cc:302] T 00000000000000000000000000000000 P 73efa44fd0214b3ab20f67187524323f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 73efa44fd0214b3ab20f67187524323f; no voters:
I20260928 05:22:22.695917 22039 leader_election.cc:288] T 00000000000000000000000000000000 P 73efa44fd0214b3ab20f67187524323f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:22:22.696004 22049 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 73efa44fd0214b3ab20f67187524323f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:22:22.696595 22049 raft_consensus.cc:686] T 00000000000000000000000000000000 P 73efa44fd0214b3ab20f67187524323f [term 1 LEADER]: Becoming Leader. State: Replica: 73efa44fd0214b3ab20f67187524323f, State: Running, Role: LEADER
I20260928 05:22:22.697172 22049 consensus_queue.cc:237] T 00000000000000000000000000000000 P 73efa44fd0214b3ab20f67187524323f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73efa44fd0214b3ab20f67187524323f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38005 } }
I20260928 05:22:22.698002 22039 sys_catalog.cc:564] T 00000000000000000000000000000000 P 73efa44fd0214b3ab20f67187524323f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:22:22.699705 22051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73efa44fd0214b3ab20f67187524323f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 73efa44fd0214b3ab20f67187524323f. Latest consensus state: current_term: 1 leader_uuid: "73efa44fd0214b3ab20f67187524323f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73efa44fd0214b3ab20f67187524323f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38005 } } }
I20260928 05:22:22.700145 22051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73efa44fd0214b3ab20f67187524323f [sys.catalog]: This master's current role is: LEADER
I20260928 05:22:22.700498 22050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73efa44fd0214b3ab20f67187524323f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "73efa44fd0214b3ab20f67187524323f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73efa44fd0214b3ab20f67187524323f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38005 } } }
I20260928 05:22:22.700819 22050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73efa44fd0214b3ab20f67187524323f [sys.catalog]: This master's current role is: LEADER
I20260928 05:22:22.703161 22056 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:22:22.708297 22056 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:22:22.716470 22056 catalog_manager.cc:1347] Generated new cluster ID: f1c9252f4bd5424fa4c741e2b4bf44f2
I20260928 05:22:22.716565 22056 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:22:22.733429 22056 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:22:22.733778 22056 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:22:22.748540 22056 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 73efa44fd0214b3ab20f67187524323f: Generated new TSK 0
I20260928 05:22:22.749487 22056 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 05:22:22.760715 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 21972
W20260928 05:22:22.761677 22034 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.143:54417
W20260928 05:22:22.762253 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 21972: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 05:22:22.762372 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 21972
2026-09-28T05:22:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1711 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790572622576160-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:22:22 dist-test-slave-whsn krb5kdc[22070](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:22:22 dist-test-slave-whsn krb5kdc[22070](info): set up 2 sockets
Sep 28 05:22:22 dist-test-slave-whsn krb5kdc[22070](info): commencing operation
krb5kdc: starting...
W20260928 05:22:23.780838 388 mini_kdc.cc:121] Time spent starting KDC: real 0.987s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790572622576160-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790572622576160-388-0/krb5kdc/test-user.keytab.
Sep 28 05:22:23 dist-test-slave-whsn krb5kdc[22070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572943, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:22:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:22:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 05:22:23.996876 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790572622576160-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790572622576160-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790572622576160-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37683
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37683
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:22:24.302645 22083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:22:24.302924 22083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:22:24.303061 22083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:22:24.309993 22083 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:22:24.310104 22083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:22:24.310187 22083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:22:24.310269 22083 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:22:24.310338 22083 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:22:24.319635 22083 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790572622576160-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790572622576160-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37683
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37683
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790572622576160-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:22:24.320817 22083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:22:24.322080 22083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:22:24.332808 22089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:22:24.332729 22088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:22:24.332844 22091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:22:24.333595 22083 server_base.cc:1034] running on GCE node
I20260928 05:22:24.334381 22083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:22:24.335640 22083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:22:24.336841 22083 hybrid_clock.cc:648] HybridClock initialized: now 1790572944336789 us; error 44 us; skew 500 ppm
I20260928 05:22:24.337198 22083 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:22:24 dist-test-slave-whsn krb5kdc[22070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572944, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:22:24.339995 22083 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 05:22:24.341318 22083 webserver.cc:460] Webserver started at http://127.0.97.62:39043/ using document root <none> and password file <none>
I20260928 05:22:24.341920 22083 fs_manager.cc:359] Metadata directory not provided
I20260928 05:22:24.342029 22083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:22:24.342299 22083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:22:24.345109 22083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790572622576160-388-0/minicluster-data/master-0/data/instance:
uuid: "bc16032fa6204990a4422a80783cf47d"
format_stamp: "Formatted at 2026-09-28 05:22:24 on dist-test-slave-whsn"
server_key: "43dc61277866bad822153b3e17e58809"
server_key_iv: "cff35ccbda3a94a2cd15a8e706efb913"
server_key_version: "encryptionkey@0"
I20260928 05:22:24.345798 22083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790572622576160-388-0/minicluster-data/master-0/wal/instance:
uuid: "bc16032fa6204990a4422a80783cf47d"
format_stamp: "Formatted at 2026-09-28 05:22:24 on dist-test-slave-whsn"
server_key: "43dc61277866bad822153b3e17e58809"
server_key_iv: "cff35ccbda3a94a2cd15a8e706efb913"
server_key_version: "encryptionkey@0"
I20260928 05:22:24.350452 22083 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 05:22:24.353588 22098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:22:24.354782 22083 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 05:22:24.354921 22083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790572622576160-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790572622576160-388-0/minicluster-data/master-0/wal
uuid: "bc16032fa6204990a4422a80783cf47d"
format_stamp: "Formatted at 2026-09-28 05:22:24 on dist-test-slave-whsn"
server_key: "43dc61277866bad822153b3e17e58809"
server_key_iv: "cff35ccbda3a94a2cd15a8e706efb913"
server_key_version: "encryptionkey@0"
I20260928 05:22:24.355087 22083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790572622576160-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790572622576160-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790572622576160-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:22:24.374648 22083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:22:24.378465 22083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:22:24.378700 22083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:22:24.394610 22083 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37683
I20260928 05:22:24.394630 22149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37683 every 8 connection(s)
I20260928 05:22:24.395895 22083 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:22:24.399529 22150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:22:24.405488 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 22083
I20260928 05:22:24.405661 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790572622576160-388-0/minicluster-data/master-0/wal/instance
I20260928 05:22:24.406179 388 external_mini_cluster.cc:1428] Setting key 69f64b0d524c90f2083f11143dcfa223
I20260928 05:22:24.409792 22150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc16032fa6204990a4422a80783cf47d: Bootstrap starting.
I20260928 05:22:24.413128 22150 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc16032fa6204990a4422a80783cf47d: Neither blocks nor log segments found. Creating new log.
I20260928 05:22:24.414112 22150 log.cc:824] T 00000000000000000000000000000000 P bc16032fa6204990a4422a80783cf47d: Log is configured to *not* fsync() on all Append() calls
I20260928 05:22:24.416828 22150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc16032fa6204990a4422a80783cf47d: No bootstrap required, opened a new log
Sep 28 05:22:24 dist-test-slave-whsn krb5kdc[22070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572943, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:22:24.426383 22150 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc16032fa6204990a4422a80783cf47d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc16032fa6204990a4422a80783cf47d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37683 } }
I20260928 05:22:24.426795 22150 raft_consensus.cc:374] T 00000000000000000000000000000000 P bc16032fa6204990a4422a80783cf47d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:22:24.426950 22150 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc16032fa6204990a4422a80783cf47d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc16032fa6204990a4422a80783cf47d, State: Initialized, Role: FOLLOWER
I20260928 05:22:24.427582 22150 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc16032fa6204990a4422a80783cf47d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc16032fa6204990a4422a80783cf47d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37683 } }
I20260928 05:22:24.427861 22150 raft_consensus.cc:388] T 00000000000000000000000000000000 P bc16032fa6204990a4422a80783cf47d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:22:24.427994 22150 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc16032fa6204990a4422a80783cf47d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:22:24.428156 22150 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc16032fa6204990a4422a80783cf47d [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:22:24.428464 22153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:24.407863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60077 (local address 127.0.97.62:37683)
0928 05:22:24.408575 (+ 712us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:24.408587 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:24.408627 (+ 40us) server_negotiation.cc:407] Connection header received
0928 05:22:24.410251 (+ 1624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:24.410273 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:24.410640 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:24.410936 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:24.412594 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:24.413955 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:24.415446 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:24.416077 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:24.416195 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:24.420583 (+ 4388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:24.420593 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:24.420609 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:22:24.420649 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:24.424385 (+ 3736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:24.425079 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:24.425087 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:24.425095 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:24.425285 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:24.425623 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:24.425628 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:24.425631 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:24.426237 (+ 606us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:24.426519 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:24.426852 (+ 333us) server_negotiation.cc:299] Negotiation successful
0928 05:22:24.427166 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":524,"thread_start_us":330,"threads_started":1}
I20260928 05:22:24.430853 22150 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc16032fa6204990a4422a80783cf47d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc16032fa6204990a4422a80783cf47d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37683 } }
I20260928 05:22:24.431365 22150 leader_election.cc:302] T 00000000000000000000000000000000 P bc16032fa6204990a4422a80783cf47d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc16032fa6204990a4422a80783cf47d; no voters:
I20260928 05:22:24.431955 22150 leader_election.cc:288] T 00000000000000000000000000000000 P bc16032fa6204990a4422a80783cf47d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:22:24.432051 22155 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc16032fa6204990a4422a80783cf47d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:22:24.432142 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 05:22:24.433756 22150 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc16032fa6204990a4422a80783cf47d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:22:24.438707 22155 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc16032fa6204990a4422a80783cf47d [term 1 LEADER]: Becoming Leader. State: Replica: bc16032fa6204990a4422a80783cf47d, State: Running, Role: LEADER
I20260928 05:22:24.439231 22155 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc16032fa6204990a4422a80783cf47d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc16032fa6204990a4422a80783cf47d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37683 } }
I20260928 05:22:24.442562 22156 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc16032fa6204990a4422a80783cf47d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bc16032fa6204990a4422a80783cf47d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc16032fa6204990a4422a80783cf47d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37683 } } }
I20260928 05:22:24.442947 22156 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc16032fa6204990a4422a80783cf47d [sys.catalog]: This master's current role is: LEADER
I20260928 05:22:24.443445 22157 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc16032fa6204990a4422a80783cf47d [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc16032fa6204990a4422a80783cf47d. Latest consensus state: current_term: 1 leader_uuid: "bc16032fa6204990a4422a80783cf47d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc16032fa6204990a4422a80783cf47d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37683 } } }
I20260928 05:22:24.443675 22157 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc16032fa6204990a4422a80783cf47d [sys.catalog]: This master's current role is: LEADER
I20260928 05:22:24.445627 22170 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:22:24.450757 22170 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:22:24.463119 22153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:24.441273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48311 (local address 127.0.97.62:37683)
0928 05:22:24.441422 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:24.441427 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:24.446171 (+ 4744us) server_negotiation.cc:407] Connection header received
0928 05:22:24.446439 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:24.446445 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:24.446521 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:24.446599 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:24.448351 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:24.449339 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:24.451330 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:24.452070 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:24.452173 (+ 103us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:24.455597 (+ 3424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:24.455604 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:24.455610 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:22:24.455651 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:24.459431 (+ 3780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:24.460421 (+ 990us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:24.460427 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:24.460431 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:24.460579 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:24.461174 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:24.461180 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:24.461183 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:24.461582 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:24.461830 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:24.462513 (+ 683us) server_negotiation.cc:299] Negotiation successful
0928 05:22:24.462704 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 05:22:24.470718 22170 catalog_manager.cc:1347] Generated new cluster ID: 895dad50871b407085e32f70f591f29e
I20260928 05:22:24.470830 22170 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:22:24.502667 22170 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:22:24.504429 22170 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:22:24.514379 22170 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc16032fa6204990a4422a80783cf47d: Generated new TSK 0
I20260928 05:22:24.515089 22170 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 05:22:24.526942 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 22083
I20260928 05:22:24.528179 22145 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:22:24.619998 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 22083
W20260928 05:22:24.633980 22068 connection.cc:535] client connection to 127.0.97.62:37683 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:37683): Connection reset by peer (error 104)
2026-09-28T05:22:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1858 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790572622576160-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:22:24 dist-test-slave-whsn krb5kdc[22183](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:22:24 dist-test-slave-whsn krb5kdc[22183](info): set up 2 sockets
Sep 28 05:22:24 dist-test-slave-whsn krb5kdc[22183](info): commencing operation
krb5kdc: starting...
W20260928 05:22:25.616931 388 mini_kdc.cc:121] Time spent starting KDC: real 0.964s user 0.002s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790572622576160-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790572622576160-388-0/krb5kdc/test-user.keytab.
Sep 28 05:22:25 dist-test-slave-whsn krb5kdc[22183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572945, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:22:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:22:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 05:22:25.824944 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790572622576160-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790572622576160-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790572622576160-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38679
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38679
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:22:26.130134 22196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:22:26.130432 22196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:22:26.130564 22196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:22:26.137534 22196 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:22:26.137648 22196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:22:26.137723 22196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:22:26.137794 22196 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:22:26.137861 22196 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:22:26.147346 22196 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790572622576160-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790572622576160-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38679
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790572622576160-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:22:26.148545 22196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:22:26.149777 22196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:22:26.160348 22204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:22:26.160406 22201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:22:26.160430 22202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:22:26.161185 22196 server_base.cc:1034] running on GCE node
I20260928 05:22:26.161897 22196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:22:26.163215 22196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:22:26.164399 22196 hybrid_clock.cc:648] HybridClock initialized: now 1790572946164352 us; error 40 us; skew 500 ppm
I20260928 05:22:26.164701 22196 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:22:26 dist-test-slave-whsn krb5kdc[22183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572946, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:22:26.167553 22196 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 05:22:26.168960 22196 webserver.cc:460] Webserver started at http://127.0.97.62:46423/ using document root <none> and password file <none>
I20260928 05:22:26.169590 22196 fs_manager.cc:359] Metadata directory not provided
I20260928 05:22:26.169698 22196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:22:26.169978 22196 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:22:26.172730 22196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790572622576160-388-0/minicluster-data/master-0/data/instance:
uuid: "ebbe2ecb8105416a89bf452fb52a9ce2"
format_stamp: "Formatted at 2026-09-28 05:22:26 on dist-test-slave-whsn"
server_key: "9c92f7311df1910c6cbcb1c82eeb364e"
server_key_iv: "3ae33432f433b781bd57026e756155b3"
server_key_version: "encryptionkey@0"
I20260928 05:22:26.173483 22196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790572622576160-388-0/minicluster-data/master-0/wal/instance:
uuid: "ebbe2ecb8105416a89bf452fb52a9ce2"
format_stamp: "Formatted at 2026-09-28 05:22:26 on dist-test-slave-whsn"
server_key: "9c92f7311df1910c6cbcb1c82eeb364e"
server_key_iv: "3ae33432f433b781bd57026e756155b3"
server_key_version: "encryptionkey@0"
I20260928 05:22:26.178257 22196 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260928 05:22:26.181425 22211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:22:26.182637 22196 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:22:26.182778 22196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790572622576160-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790572622576160-388-0/minicluster-data/master-0/wal
uuid: "ebbe2ecb8105416a89bf452fb52a9ce2"
format_stamp: "Formatted at 2026-09-28 05:22:26 on dist-test-slave-whsn"
server_key: "9c92f7311df1910c6cbcb1c82eeb364e"
server_key_iv: "3ae33432f433b781bd57026e756155b3"
server_key_version: "encryptionkey@0"
I20260928 05:22:26.182943 22196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790572622576160-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790572622576160-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790572622576160-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:22:26.198990 22196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:22:26.202502 22196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:22:26.202734 22196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:22:26.219688 22196 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38679
I20260928 05:22:26.219712 22262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38679 every 8 connection(s)
I20260928 05:22:26.221001 22196 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:22:26.222446 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 22196
I20260928 05:22:26.222596 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790572622576160-388-0/minicluster-data/master-0/wal/instance
I20260928 05:22:26.223011 388 external_mini_cluster.cc:1428] Setting key b6b8dd1b37dbbb2646969be204c11c64
I20260928 05:22:26.224839 22263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:22:26.237152 22266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:26.224643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52533 (local address 127.0.97.62:38679)
0928 05:22:26.225416 (+ 773us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:26.225430 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:26.225477 (+ 47us) server_negotiation.cc:407] Connection header received
0928 05:22:26.227558 (+ 2081us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:26.227584 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:26.228092 (+ 508us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:26.228513 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:26.229035 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:26.231124 (+ 2089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:26.232331 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:26.233338 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:26.233502 (+ 164us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:26.234543 (+ 1041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:26.234556 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:26.234573 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:22:26.235286 (+ 713us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:26.235380 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:22:26.235415 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:26.235558 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:26.235752 (+ 194us) server_negotiation.cc:299] Negotiation successful
0928 05:22:26.235928 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":324,"threads_started":1}
I20260928 05:22:26.238440 22263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebbe2ecb8105416a89bf452fb52a9ce2: Bootstrap starting.
I20260928 05:22:26.240608 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 05:22:26.241555 22263 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ebbe2ecb8105416a89bf452fb52a9ce2: Neither blocks nor log segments found. Creating new log.
I20260928 05:22:26.242666 22263 log.cc:824] T 00000000000000000000000000000000 P ebbe2ecb8105416a89bf452fb52a9ce2: Log is configured to *not* fsync() on all Append() calls
I20260928 05:22:26.245217 22263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebbe2ecb8105416a89bf452fb52a9ce2: No bootstrap required, opened a new log
I20260928 05:22:26.251525 22266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:26.245329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38721 (local address 127.0.97.62:38679)
0928 05:22:26.245480 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:26.245486 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:26.245958 (+ 472us) server_negotiation.cc:407] Connection header received
0928 05:22:26.246096 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:26.246101 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:26.246168 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:26.246274 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:26.246648 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:26.247807 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:26.249080 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:26.249768 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:26.249890 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:26.250841 (+ 951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:26.250847 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:26.250852 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:22:26.250874 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:26.250920 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:22:26.250940 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:26.251051 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:26.251158 (+ 107us) server_negotiation.cc:299] Negotiation successful
0928 05:22:26.251204 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 05:22:26.254242 22263 raft_consensus.cc:348] T 00000000000000000000000000000000 P ebbe2ecb8105416a89bf452fb52a9ce2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebbe2ecb8105416a89bf452fb52a9ce2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38679 } }
I20260928 05:22:26.254575 22263 raft_consensus.cc:374] T 00000000000000000000000000000000 P ebbe2ecb8105416a89bf452fb52a9ce2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:22:26.254667 22263 raft_consensus.cc:729] T 00000000000000000000000000000000 P ebbe2ecb8105416a89bf452fb52a9ce2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebbe2ecb8105416a89bf452fb52a9ce2, State: Initialized, Role: FOLLOWER
I20260928 05:22:26.255234 22263 consensus_queue.cc:260] T 00000000000000000000000000000000 P ebbe2ecb8105416a89bf452fb52a9ce2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebbe2ecb8105416a89bf452fb52a9ce2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38679 } }
I20260928 05:22:26.255486 22263 raft_consensus.cc:388] T 00000000000000000000000000000000 P ebbe2ecb8105416a89bf452fb52a9ce2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:22:26.255610 22263 raft_consensus.cc:482] T 00000000000000000000000000000000 P ebbe2ecb8105416a89bf452fb52a9ce2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:22:26.255755 22263 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ebbe2ecb8105416a89bf452fb52a9ce2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:22:26.257812 22263 raft_consensus.cc:504] T 00000000000000000000000000000000 P ebbe2ecb8105416a89bf452fb52a9ce2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebbe2ecb8105416a89bf452fb52a9ce2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38679 } }
I20260928 05:22:26.258234 22263 leader_election.cc:302] T 00000000000000000000000000000000 P ebbe2ecb8105416a89bf452fb52a9ce2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ebbe2ecb8105416a89bf452fb52a9ce2; no voters:
I20260928 05:22:26.258786 22263 leader_election.cc:288] T 00000000000000000000000000000000 P ebbe2ecb8105416a89bf452fb52a9ce2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:22:26.258873 22273 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ebbe2ecb8105416a89bf452fb52a9ce2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:22:26.259588 22273 raft_consensus.cc:686] T 00000000000000000000000000000000 P ebbe2ecb8105416a89bf452fb52a9ce2 [term 1 LEADER]: Becoming Leader. State: Replica: ebbe2ecb8105416a89bf452fb52a9ce2, State: Running, Role: LEADER
I20260928 05:22:26.260088 22273 consensus_queue.cc:237] T 00000000000000000000000000000000 P ebbe2ecb8105416a89bf452fb52a9ce2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebbe2ecb8105416a89bf452fb52a9ce2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38679 } }
I20260928 05:22:26.260726 22263 sys_catalog.cc:564] T 00000000000000000000000000000000 P ebbe2ecb8105416a89bf452fb52a9ce2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:22:26.262800 22274 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebbe2ecb8105416a89bf452fb52a9ce2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ebbe2ecb8105416a89bf452fb52a9ce2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebbe2ecb8105416a89bf452fb52a9ce2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38679 } } }
I20260928 05:22:26.262837 22275 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebbe2ecb8105416a89bf452fb52a9ce2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ebbe2ecb8105416a89bf452fb52a9ce2. Latest consensus state: current_term: 1 leader_uuid: "ebbe2ecb8105416a89bf452fb52a9ce2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebbe2ecb8105416a89bf452fb52a9ce2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38679 } } }
I20260928 05:22:26.263154 22274 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebbe2ecb8105416a89bf452fb52a9ce2 [sys.catalog]: This master's current role is: LEADER
I20260928 05:22:26.263171 22275 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebbe2ecb8105416a89bf452fb52a9ce2 [sys.catalog]: This master's current role is: LEADER
I20260928 05:22:26.265156 22280 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:22:26.269572 22280 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:22:26.277812 22280 catalog_manager.cc:1347] Generated new cluster ID: 22f7c0a3a58e4d01a2cb3718be132579
I20260928 05:22:26.277909 22280 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:22:26.288077 22280 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:22:26.289702 22280 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:22:26.300163 22280 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ebbe2ecb8105416a89bf452fb52a9ce2: Generated new TSK 0
I20260928 05:22:26.300987 22280 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 05:22:26.311168 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 22196
W20260928 05:22:26.312577 22258 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:52533
W20260928 05:22:26.313244 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 22196: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 05:22:26.313391 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 22196
W20260928 05:22:26.327035 22181 connection.cc:535] client connection to 127.0.97.62:38679 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:38679): Connection reset by peer (error 104)
2026-09-28T05:22:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1692 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790572622576160-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:22:26 dist-test-slave-whsn krb5kdc[22294](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:22:26 dist-test-slave-whsn krb5kdc[22294](info): set up 2 sockets
Sep 28 05:22:26 dist-test-slave-whsn krb5kdc[22294](info): commencing operation
krb5kdc: starting...
W20260928 05:22:27.321085 388 mini_kdc.cc:121] Time spent starting KDC: real 0.976s user 0.001s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790572622576160-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790572622576160-388-0/krb5kdc/test-user.keytab.
Sep 28 05:22:27 dist-test-slave-whsn krb5kdc[22294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572947, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:22:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:22:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 05:22:27.533777 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790572622576160-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790572622576160-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790572622576160-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46407
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46407
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:22:27.837997 22307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:22:27.838266 22307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:22:27.838388 22307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:22:27.845307 22307 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:22:27.845410 22307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:22:27.845482 22307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:22:27.845559 22307 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:22:27.845616 22307 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:22:27.854852 22307 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790572622576160-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790572622576160-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46407
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46407
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790572622576160-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:22:27.855962 22307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:22:27.857167 22307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:22:27.868165 22312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:22:27.868345 22313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:22:27.868572 22315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:22:27.868892 22307 server_base.cc:1034] running on GCE node
I20260928 05:22:27.869597 22307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:22:27.870827 22307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:22:27.872012 22307 hybrid_clock.cc:648] HybridClock initialized: now 1790572947871976 us; error 30 us; skew 500 ppm
I20260928 05:22:27.872313 22307 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:22:27 dist-test-slave-whsn krb5kdc[22294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572947, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:22:27.879958 22307 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 05:22:27.881279 22307 webserver.cc:460] Webserver started at http://127.0.97.62:44337/ using document root <none> and password file <none>
I20260928 05:22:27.881846 22307 fs_manager.cc:359] Metadata directory not provided
I20260928 05:22:27.881937 22307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:22:27.882159 22307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:22:27.884817 22307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790572622576160-388-0/minicluster-data/master-0/data/instance:
uuid: "29ea86a79bd14f0482bf398e7e39bcbe"
format_stamp: "Formatted at 2026-09-28 05:22:27 on dist-test-slave-whsn"
server_key: "90db3382fbd242900641da95a2148965"
server_key_iv: "7170faaaf92f02cd35b5cfb1ff90a748"
server_key_version: "encryptionkey@0"
I20260928 05:22:27.885540 22307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790572622576160-388-0/minicluster-data/master-0/wal/instance:
uuid: "29ea86a79bd14f0482bf398e7e39bcbe"
format_stamp: "Formatted at 2026-09-28 05:22:27 on dist-test-slave-whsn"
server_key: "90db3382fbd242900641da95a2148965"
server_key_iv: "7170faaaf92f02cd35b5cfb1ff90a748"
server_key_version: "encryptionkey@0"
I20260928 05:22:27.890131 22307 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260928 05:22:27.893134 22322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:22:27.894301 22307 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 05:22:27.894439 22307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790572622576160-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790572622576160-388-0/minicluster-data/master-0/wal
uuid: "29ea86a79bd14f0482bf398e7e39bcbe"
format_stamp: "Formatted at 2026-09-28 05:22:27 on dist-test-slave-whsn"
server_key: "90db3382fbd242900641da95a2148965"
server_key_iv: "7170faaaf92f02cd35b5cfb1ff90a748"
server_key_version: "encryptionkey@0"
I20260928 05:22:27.894594 22307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790572622576160-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790572622576160-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790572622576160-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:22:27.899788 22307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:22:27.903301 22307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:22:27.903541 22307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:22:27.919533 22307 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46407
I20260928 05:22:27.919571 22373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46407 every 8 connection(s)
I20260928 05:22:27.920778 22307 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:22:27.921172 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 22307
I20260928 05:22:27.921319 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790572622576160-388-0/minicluster-data/master-0/wal/instance
I20260928 05:22:27.921794 388 external_mini_cluster.cc:1428] Setting key baf119a8d1f868ba2c6bf0bf883ea34f
I20260928 05:22:27.924695 22374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:22:27.928715 22377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:27.923431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44227 (local address 127.0.97.62:46407)
0928 05:22:27.924150 (+ 719us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:27.924161 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:27.924199 (+ 38us) server_negotiation.cc:407] Connection header received
0928 05:22:27.925794 (+ 1595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:27.925816 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:27.926125 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:27.926420 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:27.927350 (+ 930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:27.927356 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:27.927367 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:22:27.927380 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:27.927454 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:22:27.927472 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:27.927623 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:27.927716 (+ 93us) server_negotiation.cc:299] Negotiation successful
0928 05:22:27.927882 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":303,"threads_started":1}
I20260928 05:22:27.932068 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 05:22:27.937037 22374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29ea86a79bd14f0482bf398e7e39bcbe: Bootstrap starting.
I20260928 05:22:27.939077 22377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:27.936871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55371 (local address 127.0.97.62:46407)
0928 05:22:27.937039 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:27.937045 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:27.937472 (+ 427us) server_negotiation.cc:407] Connection header received
0928 05:22:27.937621 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:27.937625 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:27.937669 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:27.937741 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:27.938514 (+ 773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:27.938519 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:27.938524 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:22:27.938540 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:27.938581 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:22:27.938595 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:27.938682 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:27.938780 (+ 98us) server_negotiation.cc:299] Negotiation successful
0928 05:22:27.938841 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 05:22:27.939999 22374 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 29ea86a79bd14f0482bf398e7e39bcbe: Neither blocks nor log segments found. Creating new log.
I20260928 05:22:27.941089 22374 log.cc:824] T 00000000000000000000000000000000 P 29ea86a79bd14f0482bf398e7e39bcbe: Log is configured to *not* fsync() on all Append() calls
I20260928 05:22:27.943356 22374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29ea86a79bd14f0482bf398e7e39bcbe: No bootstrap required, opened a new log
I20260928 05:22:27.950209 22374 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29ea86a79bd14f0482bf398e7e39bcbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29ea86a79bd14f0482bf398e7e39bcbe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46407 } }
I20260928 05:22:27.950507 22374 raft_consensus.cc:374] T 00000000000000000000000000000000 P 29ea86a79bd14f0482bf398e7e39bcbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:22:27.950592 22374 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29ea86a79bd14f0482bf398e7e39bcbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29ea86a79bd14f0482bf398e7e39bcbe, State: Initialized, Role: FOLLOWER
I20260928 05:22:27.951068 22374 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29ea86a79bd14f0482bf398e7e39bcbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29ea86a79bd14f0482bf398e7e39bcbe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46407 } }
I20260928 05:22:27.951252 22374 raft_consensus.cc:388] T 00000000000000000000000000000000 P 29ea86a79bd14f0482bf398e7e39bcbe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:22:27.951371 22374 raft_consensus.cc:482] T 00000000000000000000000000000000 P 29ea86a79bd14f0482bf398e7e39bcbe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:22:27.951505 22374 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29ea86a79bd14f0482bf398e7e39bcbe [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:22:27.953814 22374 raft_consensus.cc:504] T 00000000000000000000000000000000 P 29ea86a79bd14f0482bf398e7e39bcbe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29ea86a79bd14f0482bf398e7e39bcbe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46407 } }
I20260928 05:22:27.954226 22374 leader_election.cc:302] T 00000000000000000000000000000000 P 29ea86a79bd14f0482bf398e7e39bcbe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 29ea86a79bd14f0482bf398e7e39bcbe; no voters:
I20260928 05:22:27.954702 22374 leader_election.cc:288] T 00000000000000000000000000000000 P 29ea86a79bd14f0482bf398e7e39bcbe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:22:27.954814 22384 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 29ea86a79bd14f0482bf398e7e39bcbe [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:22:27.955415 22384 raft_consensus.cc:686] T 00000000000000000000000000000000 P 29ea86a79bd14f0482bf398e7e39bcbe [term 1 LEADER]: Becoming Leader. State: Replica: 29ea86a79bd14f0482bf398e7e39bcbe, State: Running, Role: LEADER
I20260928 05:22:27.955902 22384 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29ea86a79bd14f0482bf398e7e39bcbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29ea86a79bd14f0482bf398e7e39bcbe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46407 } }
I20260928 05:22:27.956413 22374 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29ea86a79bd14f0482bf398e7e39bcbe [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:22:27.958467 22385 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29ea86a79bd14f0482bf398e7e39bcbe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "29ea86a79bd14f0482bf398e7e39bcbe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29ea86a79bd14f0482bf398e7e39bcbe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46407 } } }
I20260928 05:22:27.958572 22386 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29ea86a79bd14f0482bf398e7e39bcbe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29ea86a79bd14f0482bf398e7e39bcbe. Latest consensus state: current_term: 1 leader_uuid: "29ea86a79bd14f0482bf398e7e39bcbe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29ea86a79bd14f0482bf398e7e39bcbe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46407 } } }
I20260928 05:22:27.958834 22385 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29ea86a79bd14f0482bf398e7e39bcbe [sys.catalog]: This master's current role is: LEADER
I20260928 05:22:27.958884 22386 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29ea86a79bd14f0482bf398e7e39bcbe [sys.catalog]: This master's current role is: LEADER
I20260928 05:22:27.959646 22391 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:22:27.964668 22391 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:22:27.973054 22391 catalog_manager.cc:1347] Generated new cluster ID: 30f99366ef59441caba2a8670745212d
I20260928 05:22:27.973147 22391 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:22:27.990955 22391 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:22:27.991340 22391 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:22:28.000926 22391 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 29ea86a79bd14f0482bf398e7e39bcbe: Generated new TSK 0
I20260928 05:22:28.001658 22391 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 05:22:28.015614 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 22307
W20260928 05:22:28.016553 22369 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:44227
W20260928 05:22:28.017150 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 22307: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 05:22:28.017302 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 22307
2026-09-28T05:22:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1703 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790572622576160-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:22:28 dist-test-slave-whsn krb5kdc[22405](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:22:28 dist-test-slave-whsn krb5kdc[22405](info): set up 2 sockets
Sep 28 05:22:28 dist-test-slave-whsn krb5kdc[22405](info): commencing operation
krb5kdc: starting...
W20260928 05:22:29.008828 388 mini_kdc.cc:121] Time spent starting KDC: real 0.961s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790572622576160-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790572622576160-388-0/krb5kdc/test-user.keytab.
Sep 28 05:22:29 dist-test-slave-whsn krb5kdc[22405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572949, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:22:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:22:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 05:22:29.213693 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790572622576160-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790572622576160-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790572622576160-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35043
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35043
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:22:29.516507 22418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:22:29.516767 22418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:22:29.516909 22418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:22:29.523725 22418 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:22:29.523815 22418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:22:29.523887 22418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:22:29.523968 22418 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:22:29.524024 22418 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:22:29.533242 22418 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790572622576160-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790572622576160-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35043
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35043
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790572622576160-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:22:29.534386 22418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:22:29.535579 22418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:22:29.546113 22426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:22:29.546237 22424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:22:29.546532 22423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:22:29.547134 22418 server_base.cc:1034] running on GCE node
I20260928 05:22:29.547767 22418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:22:29.548985 22418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:22:29.550175 22418 hybrid_clock.cc:648] HybridClock initialized: now 1790572949550132 us; error 37 us; skew 500 ppm
I20260928 05:22:29.550472 22418 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:22:29 dist-test-slave-whsn krb5kdc[22405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572949, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:22:29.553210 22418 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 05:22:29.554394 22418 webserver.cc:460] Webserver started at http://127.0.97.62:36617/ using document root <none> and password file <none>
I20260928 05:22:29.554960 22418 fs_manager.cc:359] Metadata directory not provided
I20260928 05:22:29.555050 22418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:22:29.555270 22418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:22:29.557938 22418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790572622576160-388-0/minicluster-data/master-0/data/instance:
uuid: "80abe6c2d140450e92be20103fbe9e87"
format_stamp: "Formatted at 2026-09-28 05:22:29 on dist-test-slave-whsn"
server_key: "046bcbda937aff8844647545a4f0e83e"
server_key_iv: "38d9b58a3ed0b06357c4c4f23571b682"
server_key_version: "encryptionkey@0"
I20260928 05:22:29.558568 22418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790572622576160-388-0/minicluster-data/master-0/wal/instance:
uuid: "80abe6c2d140450e92be20103fbe9e87"
format_stamp: "Formatted at 2026-09-28 05:22:29 on dist-test-slave-whsn"
server_key: "046bcbda937aff8844647545a4f0e83e"
server_key_iv: "38d9b58a3ed0b06357c4c4f23571b682"
server_key_version: "encryptionkey@0"
I20260928 05:22:29.563030 22418 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 05:22:29.566018 22433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:22:29.567214 22418 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 05:22:29.567373 22418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790572622576160-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790572622576160-388-0/minicluster-data/master-0/wal
uuid: "80abe6c2d140450e92be20103fbe9e87"
format_stamp: "Formatted at 2026-09-28 05:22:29 on dist-test-slave-whsn"
server_key: "046bcbda937aff8844647545a4f0e83e"
server_key_iv: "38d9b58a3ed0b06357c4c4f23571b682"
server_key_version: "encryptionkey@0"
I20260928 05:22:29.567600 22418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790572622576160-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790572622576160-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790572622576160-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:22:29.588279 22418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:22:29.591704 22418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:22:29.591944 22418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:22:29.607421 22418 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35043
I20260928 05:22:29.607455 22484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35043 every 8 connection(s)
I20260928 05:22:29.608628 22418 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:22:29.611824 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 22418
I20260928 05:22:29.611995 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790572622576160-388-0/minicluster-data/master-0/wal/instance
I20260928 05:22:29.612082 22485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:22:29.612517 388 external_mini_cluster.cc:1428] Setting key 2e41e1f0b950d5a26e4e5f6f8edac214
I20260928 05:22:29.625380 22485 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80abe6c2d140450e92be20103fbe9e87: Bootstrap starting.
Sep 28 05:22:29 dist-test-slave-whsn krb5kdc[22405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572949, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:22:29.628278 22485 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 80abe6c2d140450e92be20103fbe9e87: Neither blocks nor log segments found. Creating new log.
I20260928 05:22:29.629338 22485 log.cc:824] T 00000000000000000000000000000000 P 80abe6c2d140450e92be20103fbe9e87: Log is configured to *not* fsync() on all Append() calls
I20260928 05:22:29.631731 22485 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80abe6c2d140450e92be20103fbe9e87: No bootstrap required, opened a new log
I20260928 05:22:29.633410 22488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:29.614093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36083 (local address 127.0.97.62:35043)
0928 05:22:29.614829 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:29.614843 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:29.614887 (+ 44us) server_negotiation.cc:407] Connection header received
0928 05:22:29.616562 (+ 1675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:29.616581 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:29.616995 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:29.617317 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:29.618890 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:29.620087 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:29.621823 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:29.622478 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:29.622611 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:29.626917 (+ 4306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:29.626924 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:29.626936 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:22:29.626979 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:29.629828 (+ 2849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:29.630419 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:29.630425 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:29.630430 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:29.630585 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:29.630922 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:29.630926 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:29.630927 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:29.631411 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:29.631699 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:29.632094 (+ 395us) server_negotiation.cc:299] Negotiation successful
0928 05:22:29.632459 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":321,"threads_started":1}
I20260928 05:22:29.636672 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 05:22:29.639066 22485 raft_consensus.cc:348] T 00000000000000000000000000000000 P 80abe6c2d140450e92be20103fbe9e87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80abe6c2d140450e92be20103fbe9e87" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35043 } }
I20260928 05:22:29.639384 22485 raft_consensus.cc:374] T 00000000000000000000000000000000 P 80abe6c2d140450e92be20103fbe9e87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:22:29.639473 22485 raft_consensus.cc:729] T 00000000000000000000000000000000 P 80abe6c2d140450e92be20103fbe9e87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80abe6c2d140450e92be20103fbe9e87, State: Initialized, Role: FOLLOWER
I20260928 05:22:29.639954 22485 consensus_queue.cc:260] T 00000000000000000000000000000000 P 80abe6c2d140450e92be20103fbe9e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80abe6c2d140450e92be20103fbe9e87" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35043 } }
I20260928 05:22:29.640159 22485 raft_consensus.cc:388] T 00000000000000000000000000000000 P 80abe6c2d140450e92be20103fbe9e87 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:22:29.640278 22485 raft_consensus.cc:482] T 00000000000000000000000000000000 P 80abe6c2d140450e92be20103fbe9e87 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:22:29.640476 22485 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 80abe6c2d140450e92be20103fbe9e87 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:22:29.642504 22485 raft_consensus.cc:504] T 00000000000000000000000000000000 P 80abe6c2d140450e92be20103fbe9e87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80abe6c2d140450e92be20103fbe9e87" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35043 } }
I20260928 05:22:29.642989 22485 leader_election.cc:302] T 00000000000000000000000000000000 P 80abe6c2d140450e92be20103fbe9e87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 80abe6c2d140450e92be20103fbe9e87; no voters:
I20260928 05:22:29.643582 22485 leader_election.cc:288] T 00000000000000000000000000000000 P 80abe6c2d140450e92be20103fbe9e87 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:22:29.643852 22495 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 80abe6c2d140450e92be20103fbe9e87 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:22:29.645108 22495 raft_consensus.cc:686] T 00000000000000000000000000000000 P 80abe6c2d140450e92be20103fbe9e87 [term 1 LEADER]: Becoming Leader. State: Replica: 80abe6c2d140450e92be20103fbe9e87, State: Running, Role: LEADER
I20260928 05:22:29.645771 22485 sys_catalog.cc:564] T 00000000000000000000000000000000 P 80abe6c2d140450e92be20103fbe9e87 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:22:29.645782 22495 consensus_queue.cc:237] T 00000000000000000000000000000000 P 80abe6c2d140450e92be20103fbe9e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80abe6c2d140450e92be20103fbe9e87" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35043 } }
I20260928 05:22:29.648471 22496 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80abe6c2d140450e92be20103fbe9e87 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "80abe6c2d140450e92be20103fbe9e87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80abe6c2d140450e92be20103fbe9e87" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35043 } } }
I20260928 05:22:29.648748 22496 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80abe6c2d140450e92be20103fbe9e87 [sys.catalog]: This master's current role is: LEADER
I20260928 05:22:29.648998 22497 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80abe6c2d140450e92be20103fbe9e87 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 80abe6c2d140450e92be20103fbe9e87. Latest consensus state: current_term: 1 leader_uuid: "80abe6c2d140450e92be20103fbe9e87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80abe6c2d140450e92be20103fbe9e87" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35043 } } }
I20260928 05:22:29.649284 22497 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80abe6c2d140450e92be20103fbe9e87 [sys.catalog]: This master's current role is: LEADER
I20260928 05:22:29.656322 22503 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:22:29.658727 22488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:29.640427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46079 (local address 127.0.97.62:35043)
0928 05:22:29.640587 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:29.640592 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:29.641333 (+ 741us) server_negotiation.cc:407] Connection header received
0928 05:22:29.641474 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:29.641479 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:29.641543 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:29.641625 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:29.642925 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:29.643945 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:29.645671 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:29.646402 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:29.646529 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:29.651784 (+ 5255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:29.651792 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:29.651797 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:22:29.651846 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:29.655404 (+ 3558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:29.656205 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:29.656210 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:29.656214 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:29.656359 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:29.656817 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:29.656822 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:29.656825 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:29.657295 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:29.657578 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:29.658139 (+ 561us) server_negotiation.cc:299] Negotiation successful
0928 05:22:29.658360 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 05:22:29.662506 22503 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:22:29.669480 22512 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 05:22:29.670627 22503 catalog_manager.cc:1347] Generated new cluster ID: 94263d7fc16a4da7a15a444f36cafa76
I20260928 05:22:29.670717 22503 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:22:29.682492 22503 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:22:29.684180 22503 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:22:29.699240 22503 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 80abe6c2d140450e92be20103fbe9e87: Generated new TSK 0
I20260928 05:22:29.700204 22503 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 05:22:29.708536 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 22418
I20260928 05:22:29.709501 22480 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:22:29.797089 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 22418
2026-09-28T05:22:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1780 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790572622576160-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:22:29 dist-test-slave-whsn krb5kdc[22517](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:22:29 dist-test-slave-whsn krb5kdc[22517](info): set up 2 sockets
Sep 28 05:22:29 dist-test-slave-whsn krb5kdc[22517](info): commencing operation
krb5kdc: starting...
W20260928 05:22:30.800935 388 mini_kdc.cc:121] Time spent starting KDC: real 0.972s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790572622576160-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790572622576160-388-0/krb5kdc/test-user.keytab.
Sep 28 05:22:30 dist-test-slave-whsn krb5kdc[22517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572950, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:22:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:22:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 05:22:31.007658 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790572622576160-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790572622576160-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790572622576160-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42443
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42443
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:22:31.330830 22530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:22:31.331130 22530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:22:31.331283 22530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:22:31.338325 22530 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:22:31.338442 22530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:22:31.338531 22530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:22:31.338632 22530 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:22:31.338702 22530 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:22:31.348767 22530 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790572622576160-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790572622576160-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42443
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790572622576160-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:22:31.350023 22530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:22:31.351311 22530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:22:31.365324 22536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:22:31.365368 22535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:22:31.366585 22538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:22:31.366676 22530 server_base.cc:1034] running on GCE node
I20260928 05:22:31.367588 22530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:22:31.369309 22530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:22:31.370546 22530 hybrid_clock.cc:648] HybridClock initialized: now 1790572951370486 us; error 48 us; skew 500 ppm
I20260928 05:22:31.370963 22530 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:22:31 dist-test-slave-whsn krb5kdc[22517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572951, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:22:31.374696 22530 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 05:22:31.376361 22530 webserver.cc:460] Webserver started at http://127.0.97.62:46179/ using document root <none> and password file <none>
I20260928 05:22:31.377230 22530 fs_manager.cc:359] Metadata directory not provided
I20260928 05:22:31.377357 22530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:22:31.377667 22530 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:22:31.381688 22530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790572622576160-388-0/minicluster-data/master-0/data/instance:
uuid: "103c2ad2b99f40029561fd8e8dbff22c"
format_stamp: "Formatted at 2026-09-28 05:22:31 on dist-test-slave-whsn"
server_key: "f9926339fbd6ce7159284169a9f5407e"
server_key_iv: "f032855cc370936bbf49a6ffc3c7587a"
server_key_version: "encryptionkey@0"
I20260928 05:22:31.382637 22530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790572622576160-388-0/minicluster-data/master-0/wal/instance:
uuid: "103c2ad2b99f40029561fd8e8dbff22c"
format_stamp: "Formatted at 2026-09-28 05:22:31 on dist-test-slave-whsn"
server_key: "f9926339fbd6ce7159284169a9f5407e"
server_key_iv: "f032855cc370936bbf49a6ffc3c7587a"
server_key_version: "encryptionkey@0"
I20260928 05:22:31.389266 22530 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260928 05:22:31.393472 22545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:22:31.395012 22530 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 05:22:31.395174 22530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790572622576160-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790572622576160-388-0/minicluster-data/master-0/wal
uuid: "103c2ad2b99f40029561fd8e8dbff22c"
format_stamp: "Formatted at 2026-09-28 05:22:31 on dist-test-slave-whsn"
server_key: "f9926339fbd6ce7159284169a9f5407e"
server_key_iv: "f032855cc370936bbf49a6ffc3c7587a"
server_key_version: "encryptionkey@0"
I20260928 05:22:31.395390 22530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790572622576160-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790572622576160-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790572622576160-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:22:31.421227 22530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:22:31.424788 22530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:22:31.425045 22530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:22:31.441248 22530 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42443
I20260928 05:22:31.441280 22596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42443 every 8 connection(s)
I20260928 05:22:31.442440 22530 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:22:31.445801 22597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:22:31.446512 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 22530
I20260928 05:22:31.446691 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790572622576160-388-0/minicluster-data/master-0/wal/instance
I20260928 05:22:31.447237 388 external_mini_cluster.cc:1428] Setting key d3b84913d1fce45b73026b4383df6a54
I20260928 05:22:31.458451 22597 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 103c2ad2b99f40029561fd8e8dbff22c: Bootstrap starting.
I20260928 05:22:31.459851 22600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:31.448934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57787 (local address 127.0.97.62:42443)
0928 05:22:31.449652 (+ 718us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:31.449663 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:31.449701 (+ 38us) server_negotiation.cc:407] Connection header received
0928 05:22:31.451646 (+ 1945us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:31.451666 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:31.452048 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:31.452351 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:31.452874 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:31.454069 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:31.455845 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:31.456502 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:31.456628 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:31.457797 (+ 1169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:31.457805 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:31.457817 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:22:31.458315 (+ 498us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:31.458384 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:22:31.458406 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:31.458491 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:31.458716 (+ 225us) server_negotiation.cc:299] Negotiation successful
0928 05:22:31.458940 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":303,"threads_started":1}
I20260928 05:22:31.461609 22597 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 103c2ad2b99f40029561fd8e8dbff22c: Neither blocks nor log segments found. Creating new log.
I20260928 05:22:31.462728 22597 log.cc:824] T 00000000000000000000000000000000 P 103c2ad2b99f40029561fd8e8dbff22c: Log is configured to *not* fsync() on all Append() calls
I20260928 05:22:31.463227 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 05:22:31.465721 22597 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 103c2ad2b99f40029561fd8e8dbff22c: No bootstrap required, opened a new log
I20260928 05:22:31.473628 22600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:31.467688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50169 (local address 127.0.97.62:42443)
0928 05:22:31.467846 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:31.467852 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:31.468324 (+ 472us) server_negotiation.cc:407] Connection header received
0928 05:22:31.468443 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:31.468448 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:31.468525 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:31.468624 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:31.469040 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:31.470078 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:31.471241 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:31.471929 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:31.472050 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:31.472936 (+ 886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:31.472943 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:31.472947 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:22:31.472970 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:31.473039 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:22:31.473060 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:31.473150 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:31.473231 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 05:22:31.473319 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 05:22:31.475023 22597 raft_consensus.cc:348] T 00000000000000000000000000000000 P 103c2ad2b99f40029561fd8e8dbff22c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "103c2ad2b99f40029561fd8e8dbff22c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42443 } }
I20260928 05:22:31.475329 22597 raft_consensus.cc:374] T 00000000000000000000000000000000 P 103c2ad2b99f40029561fd8e8dbff22c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:22:31.475402 22597 raft_consensus.cc:729] T 00000000000000000000000000000000 P 103c2ad2b99f40029561fd8e8dbff22c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 103c2ad2b99f40029561fd8e8dbff22c, State: Initialized, Role: FOLLOWER
I20260928 05:22:31.475880 22597 consensus_queue.cc:260] T 00000000000000000000000000000000 P 103c2ad2b99f40029561fd8e8dbff22c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "103c2ad2b99f40029561fd8e8dbff22c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42443 } }
I20260928 05:22:31.476078 22597 raft_consensus.cc:388] T 00000000000000000000000000000000 P 103c2ad2b99f40029561fd8e8dbff22c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:22:31.476167 22597 raft_consensus.cc:482] T 00000000000000000000000000000000 P 103c2ad2b99f40029561fd8e8dbff22c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:22:31.476277 22597 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 103c2ad2b99f40029561fd8e8dbff22c [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:22:31.478224 22597 raft_consensus.cc:504] T 00000000000000000000000000000000 P 103c2ad2b99f40029561fd8e8dbff22c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "103c2ad2b99f40029561fd8e8dbff22c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42443 } }
I20260928 05:22:31.478621 22597 leader_election.cc:302] T 00000000000000000000000000000000 P 103c2ad2b99f40029561fd8e8dbff22c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 103c2ad2b99f40029561fd8e8dbff22c; no voters:
I20260928 05:22:31.479089 22597 leader_election.cc:288] T 00000000000000000000000000000000 P 103c2ad2b99f40029561fd8e8dbff22c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:22:31.479204 22607 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 103c2ad2b99f40029561fd8e8dbff22c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:22:31.479821 22607 raft_consensus.cc:686] T 00000000000000000000000000000000 P 103c2ad2b99f40029561fd8e8dbff22c [term 1 LEADER]: Becoming Leader. State: Replica: 103c2ad2b99f40029561fd8e8dbff22c, State: Running, Role: LEADER
I20260928 05:22:31.480378 22607 consensus_queue.cc:237] T 00000000000000000000000000000000 P 103c2ad2b99f40029561fd8e8dbff22c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "103c2ad2b99f40029561fd8e8dbff22c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42443 } }
I20260928 05:22:31.481010 22597 sys_catalog.cc:564] T 00000000000000000000000000000000 P 103c2ad2b99f40029561fd8e8dbff22c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:22:31.483412 22609 sys_catalog.cc:455] T 00000000000000000000000000000000 P 103c2ad2b99f40029561fd8e8dbff22c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 103c2ad2b99f40029561fd8e8dbff22c. Latest consensus state: current_term: 1 leader_uuid: "103c2ad2b99f40029561fd8e8dbff22c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "103c2ad2b99f40029561fd8e8dbff22c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42443 } } }
I20260928 05:22:31.483438 22608 sys_catalog.cc:455] T 00000000000000000000000000000000 P 103c2ad2b99f40029561fd8e8dbff22c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "103c2ad2b99f40029561fd8e8dbff22c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "103c2ad2b99f40029561fd8e8dbff22c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42443 } } }
I20260928 05:22:31.483781 22609 sys_catalog.cc:458] T 00000000000000000000000000000000 P 103c2ad2b99f40029561fd8e8dbff22c [sys.catalog]: This master's current role is: LEADER
I20260928 05:22:31.483807 22608 sys_catalog.cc:458] T 00000000000000000000000000000000 P 103c2ad2b99f40029561fd8e8dbff22c [sys.catalog]: This master's current role is: LEADER
I20260928 05:22:31.490226 22615 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:22:31.495003 22615 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:22:31.503352 22615 catalog_manager.cc:1347] Generated new cluster ID: 44ba61eacd814cf28fba212ad3418a3b
I20260928 05:22:31.503451 22615 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:22:31.553556 22615 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:22:31.554852 22615 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:22:31.564584 22615 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 103c2ad2b99f40029561fd8e8dbff22c: Generated new TSK 0
I20260928 05:22:31.565383 22615 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 05:22:31.576246 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 22530
W20260928 05:22:31.577521 22592 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:57787
W20260928 05:22:31.578111 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 22530: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 05:22:31.578223 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 22530
2026-09-28T05:22:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1780 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790572622576160-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:22:31 dist-test-slave-whsn krb5kdc[22628](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:22:31 dist-test-slave-whsn krb5kdc[22628](info): set up 2 sockets
Sep 28 05:22:31 dist-test-slave-whsn krb5kdc[22628](info): commencing operation
krb5kdc: starting...
W20260928 05:22:32.580921 388 mini_kdc.cc:121] Time spent starting KDC: real 0.972s user 0.001s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790572622576160-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790572622576160-388-0/krb5kdc/test-user.keytab.
Sep 28 05:22:32 dist-test-slave-whsn krb5kdc[22628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572952, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:22:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:22:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 05:22:32.791008 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790572622576160-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790572622576160-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790572622576160-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39859
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39859
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:22:33.097549 22641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:22:33.097802 22641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:22:33.097898 22641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:22:33.104904 22641 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:22:33.105007 22641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:22:33.105058 22641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:22:33.105118 22641 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:22:33.105165 22641 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:22:33.114781 22641 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790572622576160-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790572622576160-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39859
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790572622576160-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:22:33.115926 22641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:22:33.117198 22641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:22:33.127863 22647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:22:33.127818 22649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:22:33.127902 22646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:22:33.128611 22641 server_base.cc:1034] running on GCE node
I20260928 05:22:33.129364 22641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:22:33.130669 22641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:22:33.131886 22641 hybrid_clock.cc:648] HybridClock initialized: now 1790572953131836 us; error 45 us; skew 500 ppm
I20260928 05:22:33.132234 22641 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:22:33 dist-test-slave-whsn krb5kdc[22628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572953, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:22:33.139377 22641 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 05:22:33.140671 22641 webserver.cc:460] Webserver started at http://127.0.97.62:42257/ using document root <none> and password file <none>
I20260928 05:22:33.141332 22641 fs_manager.cc:359] Metadata directory not provided
I20260928 05:22:33.141431 22641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:22:33.141700 22641 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:22:33.144580 22641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790572622576160-388-0/minicluster-data/master-0/data/instance:
uuid: "f8e2c776196b4d22bf8c96028cea6a25"
format_stamp: "Formatted at 2026-09-28 05:22:33 on dist-test-slave-whsn"
server_key: "bec3d1c1a2592fa81cfe5bb3d3de06af"
server_key_iv: "4d22ee0f742be6b040b6e35206c1eb7d"
server_key_version: "encryptionkey@0"
I20260928 05:22:33.145318 22641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790572622576160-388-0/minicluster-data/master-0/wal/instance:
uuid: "f8e2c776196b4d22bf8c96028cea6a25"
format_stamp: "Formatted at 2026-09-28 05:22:33 on dist-test-slave-whsn"
server_key: "bec3d1c1a2592fa81cfe5bb3d3de06af"
server_key_iv: "4d22ee0f742be6b040b6e35206c1eb7d"
server_key_version: "encryptionkey@0"
I20260928 05:22:33.150094 22641 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260928 05:22:33.153221 22656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:22:33.154366 22641 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.002s
I20260928 05:22:33.154507 22641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790572622576160-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790572622576160-388-0/minicluster-data/master-0/wal
uuid: "f8e2c776196b4d22bf8c96028cea6a25"
format_stamp: "Formatted at 2026-09-28 05:22:33 on dist-test-slave-whsn"
server_key: "bec3d1c1a2592fa81cfe5bb3d3de06af"
server_key_iv: "4d22ee0f742be6b040b6e35206c1eb7d"
server_key_version: "encryptionkey@0"
I20260928 05:22:33.154666 22641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790572622576160-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790572622576160-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790572622576160-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:22:33.159731 22641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:22:33.163193 22641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:22:33.163436 22641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:22:33.179804 22641 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39859
I20260928 05:22:33.179836 22707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39859 every 8 connection(s)
I20260928 05:22:33.181098 22641 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:22:33.184438 22708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:22:33.188339 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 22641
I20260928 05:22:33.188509 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790572622576160-388-0/minicluster-data/master-0/wal/instance
I20260928 05:22:33.189043 388 external_mini_cluster.cc:1428] Setting key 94e9fbeb8873058236d47199f9f42c85
I20260928 05:22:33.195847 22711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:33.190622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37293 (local address 127.0.97.62:39859)
0928 05:22:33.191307 (+ 685us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:33.191318 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:33.191355 (+ 37us) server_negotiation.cc:407] Connection header received
0928 05:22:33.192842 (+ 1487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:33.192891 (+ 49us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:33.193216 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:33.193516 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:33.194467 (+ 951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:33.194473 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:33.194484 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:22:33.194497 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:33.194553 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:22:33.194571 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:33.194727 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:33.194839 (+ 112us) server_negotiation.cc:299] Negotiation successful
0928 05:22:33.195015 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":297,"threads_started":1}
I20260928 05:22:33.196457 22708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8e2c776196b4d22bf8c96028cea6a25: Bootstrap starting.
I20260928 05:22:33.199280 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 05:22:33.199808 22708 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f8e2c776196b4d22bf8c96028cea6a25: Neither blocks nor log segments found. Creating new log.
I20260928 05:22:33.200975 22708 log.cc:824] T 00000000000000000000000000000000 P f8e2c776196b4d22bf8c96028cea6a25: Log is configured to *not* fsync() on all Append() calls
I20260928 05:22:33.203897 22708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8e2c776196b4d22bf8c96028cea6a25: No bootstrap required, opened a new log
I20260928 05:22:33.206060 22711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:33.203840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54679 (local address 127.0.97.62:39859)
0928 05:22:33.204001 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:33.204006 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:33.204382 (+ 376us) server_negotiation.cc:407] Connection header received
0928 05:22:33.204501 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:33.204506 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:33.204560 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:33.204668 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:33.205468 (+ 800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:33.205477 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:33.205482 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:22:33.205499 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:33.205545 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:22:33.205568 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:33.205680 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:33.205724 (+ 44us) server_negotiation.cc:299] Negotiation successful
0928 05:22:33.205774 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 05:22:33.211647 22708 raft_consensus.cc:348] T 00000000000000000000000000000000 P f8e2c776196b4d22bf8c96028cea6a25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8e2c776196b4d22bf8c96028cea6a25" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39859 } }
I20260928 05:22:33.211984 22708 raft_consensus.cc:374] T 00000000000000000000000000000000 P f8e2c776196b4d22bf8c96028cea6a25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:22:33.212052 22708 raft_consensus.cc:729] T 00000000000000000000000000000000 P f8e2c776196b4d22bf8c96028cea6a25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8e2c776196b4d22bf8c96028cea6a25, State: Initialized, Role: FOLLOWER
I20260928 05:22:33.212519 22708 consensus_queue.cc:260] T 00000000000000000000000000000000 P f8e2c776196b4d22bf8c96028cea6a25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8e2c776196b4d22bf8c96028cea6a25" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39859 } }
I20260928 05:22:33.212723 22708 raft_consensus.cc:388] T 00000000000000000000000000000000 P f8e2c776196b4d22bf8c96028cea6a25 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:22:33.212874 22708 raft_consensus.cc:482] T 00000000000000000000000000000000 P f8e2c776196b4d22bf8c96028cea6a25 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:22:33.213048 22708 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f8e2c776196b4d22bf8c96028cea6a25 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:22:33.215153 22708 raft_consensus.cc:504] T 00000000000000000000000000000000 P f8e2c776196b4d22bf8c96028cea6a25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8e2c776196b4d22bf8c96028cea6a25" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39859 } }
I20260928 05:22:33.215572 22708 leader_election.cc:302] T 00000000000000000000000000000000 P f8e2c776196b4d22bf8c96028cea6a25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f8e2c776196b4d22bf8c96028cea6a25; no voters:
I20260928 05:22:33.216059 22708 leader_election.cc:288] T 00000000000000000000000000000000 P f8e2c776196b4d22bf8c96028cea6a25 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:22:33.216163 22718 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f8e2c776196b4d22bf8c96028cea6a25 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:22:33.216751 22718 raft_consensus.cc:686] T 00000000000000000000000000000000 P f8e2c776196b4d22bf8c96028cea6a25 [term 1 LEADER]: Becoming Leader. State: Replica: f8e2c776196b4d22bf8c96028cea6a25, State: Running, Role: LEADER
I20260928 05:22:33.217319 22718 consensus_queue.cc:237] T 00000000000000000000000000000000 P f8e2c776196b4d22bf8c96028cea6a25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8e2c776196b4d22bf8c96028cea6a25" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39859 } }
I20260928 05:22:33.217967 22708 sys_catalog.cc:564] T 00000000000000000000000000000000 P f8e2c776196b4d22bf8c96028cea6a25 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:22:33.219728 22720 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8e2c776196b4d22bf8c96028cea6a25 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8e2c776196b4d22bf8c96028cea6a25. Latest consensus state: current_term: 1 leader_uuid: "f8e2c776196b4d22bf8c96028cea6a25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8e2c776196b4d22bf8c96028cea6a25" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39859 } } }
I20260928 05:22:33.220163 22720 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8e2c776196b4d22bf8c96028cea6a25 [sys.catalog]: This master's current role is: LEADER
I20260928 05:22:33.220718 22719 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8e2c776196b4d22bf8c96028cea6a25 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f8e2c776196b4d22bf8c96028cea6a25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8e2c776196b4d22bf8c96028cea6a25" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39859 } } }
I20260928 05:22:33.221042 22719 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8e2c776196b4d22bf8c96028cea6a25 [sys.catalog]: This master's current role is: LEADER
I20260928 05:22:33.221306 22726 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:22:33.231673 22726 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:22:33.239691 22726 catalog_manager.cc:1347] Generated new cluster ID: 8422b8e0f7b74ff0865277b696f2d5fb
I20260928 05:22:33.239784 22726 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:22:33.261440 22726 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:22:33.261891 22726 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:22:33.273407 22726 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f8e2c776196b4d22bf8c96028cea6a25: Generated new TSK 0
I20260928 05:22:33.274304 22726 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 05:22:33.283058 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 22641
W20260928 05:22:33.284019 22703 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:37293
W20260928 05:22:33.284686 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 22641: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 05:22:33.284816 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 22641
2026-09-28T05:22:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1707 ms)
[----------] 18 tests from AuthTokenIssuingTest (31551 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572622576160-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:22:33 dist-test-slave-whsn krb5kdc[22739](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:22:33 dist-test-slave-whsn krb5kdc[22739](info): set up 2 sockets
Sep 28 05:22:33 dist-test-slave-whsn krb5kdc[22739](info): commencing operation
krb5kdc: starting...
W20260928 05:22:34.289260 388 mini_kdc.cc:121] Time spent starting KDC: real 0.972s user 0.009s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572622576160-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572622576160-388-0/krb5kdc/test-user.keytab.
Sep 28 05:22:34 dist-test-slave-whsn krb5kdc[22739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572954, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:22:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:22:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 05:22:34.514213 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572622576160-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572622576160-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572622576160-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44509
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44509,127.0.97.61:41089,127.0.97.60:46227
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:22:34.821475 22752 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 05:22:34.821826 22752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:22:34.821928 22752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:22:34.822057 22752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:22:34.829351 22752 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:22:34.829429 22752 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:22:34.829489 22752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:22:34.829555 22752 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:22:34.829635 22752 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:22:34.839133 22752 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572622576160-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572622576160-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44509,127.0.97.61:41089,127.0.97.60:46227
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572622576160-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:22:34.840340 22752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:22:34.841642 22752 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:22:34.852352 22758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:22:34.852388 22760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:22:34.852620 22757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:22:34.853415 22752 server_base.cc:1034] running on GCE node
I20260928 05:22:34.854200 22752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:22:34.855516 22752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:22:34.856741 22752 hybrid_clock.cc:648] HybridClock initialized: now 1790572954856674 us; error 33 us; skew 500 ppm
I20260928 05:22:34.857069 22752 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:22:34 dist-test-slave-whsn krb5kdc[22739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572954, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:22:34.859863 22752 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 05:22:34.861169 22752 webserver.cc:460] Webserver started at http://127.0.97.62:37307/ using document root <none> and password file <none>
I20260928 05:22:34.861799 22752 fs_manager.cc:359] Metadata directory not provided
I20260928 05:22:34.861894 22752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:22:34.862177 22752 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:22:34.864974 22752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572622576160-388-0/minicluster-data/master-0/data/instance:
uuid: "ea47b77ae9dc44a2b760efd9ed254d9b"
format_stamp: "Formatted at 2026-09-28 05:22:34 on dist-test-slave-whsn"
server_key: "a026a48c72ba9687795c60de98585b1c"
server_key_iv: "c4a06f429a0e4edd9847b85d5df2802f"
server_key_version: "encryptionkey@0"
I20260928 05:22:34.865707 22752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572622576160-388-0/minicluster-data/master-0/wal/instance:
uuid: "ea47b77ae9dc44a2b760efd9ed254d9b"
format_stamp: "Formatted at 2026-09-28 05:22:34 on dist-test-slave-whsn"
server_key: "a026a48c72ba9687795c60de98585b1c"
server_key_iv: "c4a06f429a0e4edd9847b85d5df2802f"
server_key_version: "encryptionkey@0"
I20260928 05:22:34.870443 22752 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 05:22:34.873634 22767 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:22:34.874797 22752 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 05:22:34.874948 22752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572622576160-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572622576160-388-0/minicluster-data/master-0/wal
uuid: "ea47b77ae9dc44a2b760efd9ed254d9b"
format_stamp: "Formatted at 2026-09-28 05:22:34 on dist-test-slave-whsn"
server_key: "a026a48c72ba9687795c60de98585b1c"
server_key_iv: "c4a06f429a0e4edd9847b85d5df2802f"
server_key_version: "encryptionkey@0"
I20260928 05:22:34.875124 22752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572622576160-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572622576160-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572622576160-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:22:34.889196 22752 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:22:34.892658 22752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:22:34.892901 22752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:22:34.909462 22752 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44509
I20260928 05:22:34.909483 22818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44509 every 8 connection(s)
I20260928 05:22:34.910708 22752 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:22:34.913554 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 22752
I20260928 05:22:34.913769 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572622576160-388-0/minicluster-data/master-0/wal/instance
I20260928 05:22:34.914546 388 external_mini_cluster.cc:1428] Setting key 8a0c8ea65890bcad53764af4b2727136
I20260928 05:22:34.914654 22819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:22:34.922726 22819 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44509 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 05:22:34 dist-test-slave-whsn krb5kdc[22739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572954, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260928 05:22:34.944720 22822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:34.926188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35460 (local address 127.0.97.62:44509)
0928 05:22:34.926974 (+ 786us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:34.926987 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:34.927519 (+ 532us) server_negotiation.cc:407] Connection header received
0928 05:22:34.929421 (+ 1902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:34.929427 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:34.929696 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:34.929903 (+ 207us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:34.930989 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:34.932630 (+ 1641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:34.934033 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:34.934853 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:34.935023 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:34.938127 (+ 3104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:34.938137 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:34.938152 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:22:34.938186 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:34.940980 (+ 2794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:34.941690 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:34.941698 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:34.941701 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:34.941881 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:34.942314 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:34.942317 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:34.942319 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:34.942747 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:34.942890 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:34.943495 (+ 605us) server_negotiation.cc:299] Negotiation successful
0928 05:22:34.943722 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":539,"thread_start_us":320,"threads_started":1}
I20260928 05:22:34.944761 22823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:34.926188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44509 (local address 127.0.0.1:35460)
0928 05:22:34.927341 (+ 1153us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:34.927392 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 05:22:34.929076 (+ 1684us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:22:34.929904 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:22:34.929909 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:22:34.930455 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:22:34.930777 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:34.930796 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:34.932782 (+ 1986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:34.932786 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:34.933902 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:34.933914 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:34.935064 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:34.935069 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:34.935261 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:34.935838 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:22:34.935859 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:22:34.937899 (+ 2040us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:22:34.941110 (+ 3211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:22:34.941120 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:22:34.941127 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:22:34.941565 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:22:34.941990 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:22:34.941995 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:22:34.941999 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:22:34.942223 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:22:34.942881 (+ 658us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:22:34.942889 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:22:34.943133 (+ 244us) client_negotiation.cc:769] Sending connection context
0928 05:22:34.943384 (+ 251us) client_negotiation.cc:240] Negotiation successful
0928 05:22:34.943883 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":901,"mutex_wait_us":126,"thread_start_us":692,"threads_started":1}
I20260928 05:22:34.947494 22819 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41089 } has no permanent_uuid. Determining permanent_uuid...
I20260928 05:22:34.948402 22823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:34.948015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41089 (local address 127.0.0.1:56204)
0928 05:22:34.948180 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:34.948291 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260928 05:22:34.948618 22768 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41089: connect: Connection refused (error 111)
I20260928 05:22:34.950038 22823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:34.949668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41089 (local address 127.0.0.1:56216)
0928 05:22:34.949816 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:34.949895 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260928 05:22:34.950228 22819 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:41089: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41089: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572622576160-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572622576160-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260928 05:22:34.967392 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572622576160-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572622576160-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572622576160-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572622576160-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:41089
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572622576160-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44509,127.0.97.61:41089,127.0.97.60:46227
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 05:22:34.999502 22819 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41089 } attempt: 1
I20260928 05:22:35.000617 22823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:35.000197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41089 (local address 127.0.0.1:56228)
0928 05:22:35.000374 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:35.000457 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260928 05:22:35.001523 22823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:35.001215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41089 (local address 127.0.0.1:56244)
0928 05:22:35.001340 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:35.001404 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260928 05:22:35.001701 22819 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:41089: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41089: connect: Connection refused (error 111)
I20260928 05:22:35.069896 22819 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41089 } attempt: 2
I20260928 05:22:35.071033 22823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:35.070588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41089 (local address 127.0.0.1:56248)
0928 05:22:35.070814 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:35.070899 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260928 05:22:35.071995 22823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:35.071647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41089 (local address 127.0.0.1:56252)
0928 05:22:35.071795 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:35.071874 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260928 05:22:35.072178 22819 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:41089: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41089: connect: Connection refused (error 111)
I20260928 05:22:35.163389 22819 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41089 } attempt: 3
I20260928 05:22:35.164392 22823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:35.163977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41089 (local address 127.0.0.1:56258)
0928 05:22:35.164166 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:35.164249 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260928 05:22:35.165320 22823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:35.164987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41089 (local address 127.0.0.1:56264)
0928 05:22:35.165121 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:35.165191 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260928 05:22:35.165483 22819 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:41089: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41089: connect: Connection refused (error 111)
W20260928 05:22:35.271025 22826 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 05:22:35.271344 22826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:22:35.271437 22826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:22:35.271544 22826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:22:35.278609 22826 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:22:35.278700 22826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:22:35.278787 22826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:22:35.278851 22826 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:22:35.278939 22826 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:22:35.288478 22826 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572622576160-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572622576160-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44509,127.0.97.61:41089,127.0.97.60:46227
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572622576160-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:41089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572622576160-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572622576160-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:22:35.289693 22826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:22:35.290922 22826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:22:35.301378 22831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:22:35.301522 22834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:22:35.301542 22832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:22:35.302205 22826 server_base.cc:1034] running on GCE node
I20260928 05:22:35.302856 22826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:22:35.304090 22826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:22:35.305315 22826 hybrid_clock.cc:648] HybridClock initialized: now 1790572955305243 us; error 49 us; skew 500 ppm
I20260928 05:22:35.305574 22826 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:22:35 dist-test-slave-whsn krb5kdc[22739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572955, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:22:35.308696 22819 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41089 } attempt: 4
I20260928 05:22:35.309288 22826 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260928 05:22:35.310595 22823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:35.309756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41089 (local address 127.0.0.1:56278)
0928 05:22:35.310202 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:35.310351 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251}
I20260928 05:22:35.312716 22826 webserver.cc:460] Webserver started at http://127.0.97.61:45895/ using document root <none> and password file <none>
I20260928 05:22:35.313198 22823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:35.311929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41089 (local address 127.0.0.1:56280)
0928 05:22:35.312937 (+ 1008us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:35.313031 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":885}
I20260928 05:22:35.313649 22826 fs_manager.cc:359] Metadata directory not provided
I20260928 05:22:35.313791 22826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
W20260928 05:22:35.313807 22819 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:41089: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41089: connect: Connection refused (error 111)
I20260928 05:22:35.314184 22826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:22:35.318288 22826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572622576160-388-0/minicluster-data/master-1/data/instance:
uuid: "7224764aaf5c4cfeb98f8dce2b2e47fb"
format_stamp: "Formatted at 2026-09-28 05:22:35 on dist-test-slave-whsn"
server_key: "06e31183d40b4dc3631c14975cb3cd5f"
server_key_iv: "9d56625d0a78f9244e981555ec3ae4e2"
server_key_version: "encryptionkey@0"
I20260928 05:22:35.319334 22826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572622576160-388-0/minicluster-data/master-1/wal/instance:
uuid: "7224764aaf5c4cfeb98f8dce2b2e47fb"
format_stamp: "Formatted at 2026-09-28 05:22:35 on dist-test-slave-whsn"
server_key: "06e31183d40b4dc3631c14975cb3cd5f"
server_key_iv: "9d56625d0a78f9244e981555ec3ae4e2"
server_key_version: "encryptionkey@0"
I20260928 05:22:35.325889 22826 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260928 05:22:35.330305 22841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:22:35.331910 22826 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 05:22:35.332064 22826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572622576160-388-0/minicluster-data/master-1/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572622576160-388-0/minicluster-data/master-1/wal
uuid: "7224764aaf5c4cfeb98f8dce2b2e47fb"
format_stamp: "Formatted at 2026-09-28 05:22:35 on dist-test-slave-whsn"
server_key: "06e31183d40b4dc3631c14975cb3cd5f"
server_key_iv: "9d56625d0a78f9244e981555ec3ae4e2"
server_key_version: "encryptionkey@0"
I20260928 05:22:35.332258 22826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572622576160-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572622576160-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572622576160-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:22:35.360276 22826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:22:35.364246 22826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:22:35.364475 22826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:22:35.380445 22826 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:41089
I20260928 05:22:35.380453 22892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:41089 every 8 connection(s)
I20260928 05:22:35.381731 22826 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572622576160-388-0/minicluster-data/master-1/data/info.pb
I20260928 05:22:35.385177 22893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:22:35.385456 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 22826
I20260928 05:22:35.385607 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572622576160-388-0/minicluster-data/master-1/wal/instance
I20260928 05:22:35.385989 388 external_mini_cluster.cc:1428] Setting key 2cc93ba9fe2167e949363ebd7699e775
I20260928 05:22:35.392575 22893 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44509 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 05:22:35 dist-test-slave-whsn krb5kdc[22739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572955, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
I20260928 05:22:35.414434 22822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:35.396293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35476 (local address 127.0.97.62:44509)
0928 05:22:35.396486 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:35.396494 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:35.397641 (+ 1147us) server_negotiation.cc:407] Connection header received
0928 05:22:35.400225 (+ 2584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:35.400232 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:35.400331 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:35.400402 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:35.402349 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:35.403280 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:35.404687 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:35.405234 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:35.405357 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:35.408809 (+ 3452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:35.408818 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:35.408824 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:22:35.408897 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:35.411641 (+ 2744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:35.412481 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:35.412485 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:35.412487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:35.412605 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:35.412985 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:35.412989 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:35.412992 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:35.413274 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:35.413390 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:35.414022 (+ 632us) server_negotiation.cc:299] Negotiation successful
0928 05:22:35.414189 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
I20260928 05:22:35.415364 22896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:35.396646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44509 (local address 127.0.0.1:35476)
0928 05:22:35.397491 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:35.397539 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 05:22:35.399904 (+ 2365us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:22:35.400929 (+ 1025us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:22:35.400940 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:22:35.401626 (+ 686us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:22:35.402175 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:35.402192 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:35.403423 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:35.403426 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:35.404556 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:35.404568 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:35.405409 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:35.405414 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:35.405602 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:35.406200 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:22:35.406220 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:22:35.408566 (+ 2346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:22:35.411785 (+ 3219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:22:35.411795 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:22:35.411810 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:22:35.412357 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:22:35.412691 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:22:35.412694 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:22:35.412697 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:22:35.412898 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:22:35.413371 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:22:35.413376 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:22:35.413671 (+ 295us) client_negotiation.cc:769] Sending connection context
0928 05:22:35.413926 (+ 255us) client_negotiation.cc:240] Negotiation successful
0928 05:22:35.414298 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":363,"threads_started":1}
I20260928 05:22:35.417052 22893 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41089 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 05:22:35 dist-test-slave-whsn krb5kdc[22739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572955, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260928 05:22:35.430653 22896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:35.417754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41089 (local address 127.0.0.1:56282)
0928 05:22:35.417923 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:35.417940 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 05:22:35.418029 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:22:35.419019 (+ 990us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:22:35.419023 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:22:35.419384 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:22:35.419532 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:35.419541 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:35.421289 (+ 1748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:35.421293 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:35.422091 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:35.422101 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:35.422992 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:35.423002 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:35.423169 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:35.423666 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:22:35.423687 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:22:35.425681 (+ 1994us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:22:35.428397 (+ 2716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:22:35.428402 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:22:35.428406 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:22:35.428734 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:22:35.429127 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:22:35.429131 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:22:35.429135 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:22:35.429256 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:22:35.429762 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:22:35.429771 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:22:35.429937 (+ 166us) client_negotiation.cc:769] Sending connection context
0928 05:22:35.430047 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 05:22:35.430284 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260928 05:22:35.430753 22897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:35.417977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56282 (local address 127.0.97.61:41089)
0928 05:22:35.418580 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:35.418587 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:35.418613 (+ 26us) server_negotiation.cc:407] Connection header received
0928 05:22:35.418661 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:35.418668 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:35.418879 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:35.419009 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:35.419693 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:35.421151 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:35.422219 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:35.422861 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:35.422951 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:35.425858 (+ 2907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:35.425868 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:35.425879 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:22:35.425905 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:35.428287 (+ 2382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:35.428843 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:35.428925 (+ 82us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:35.428927 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:35.429038 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:35.429350 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:35.429355 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:35.429358 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:35.429652 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:35.429842 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:35.430191 (+ 349us) server_negotiation.cc:299] Negotiation successful
0928 05:22:35.430400 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":320,"threads_started":1}
I20260928 05:22:35.433139 22893 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46227 } has no permanent_uuid. Determining permanent_uuid...
I20260928 05:22:35.434098 22896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:35.433692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46227 (local address 127.0.0.1:51254)
0928 05:22:35.433852 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:35.433950 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260928 05:22:35.434306 22842 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46227: connect: Connection refused (error 111)
I20260928 05:22:35.435627 22896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:35.435335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46227 (local address 127.0.0.1:51266)
0928 05:22:35.435463 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:35.435532 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260928 05:22:35.435813 22893 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46227: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46227: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572622576160-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572622576160-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260928 05:22:35.441861 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572622576160-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572622576160-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572622576160-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572622576160-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:46227
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572622576160-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44509,127.0.97.61:41089,127.0.97.60:46227
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 05:22:35.485071 22893 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46227 } attempt: 1
I20260928 05:22:35.486080 22896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:35.485680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46227 (local address 127.0.0.1:51274)
0928 05:22:35.485854 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:35.485938 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260928 05:22:35.487006 22896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:35.486677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46227 (local address 127.0.0.1:51286)
0928 05:22:35.486811 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:35.486903 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260928 05:22:35.487279 22893 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46227: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46227: connect: Connection refused (error 111)
I20260928 05:22:35.555467 22893 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46227 } attempt: 2
I20260928 05:22:35.556494 22896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:35.556067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46227 (local address 127.0.0.1:51294)
0928 05:22:35.556250 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:35.556343 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260928 05:22:35.557477 22896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:35.557182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46227 (local address 127.0.0.1:51304)
0928 05:22:35.557304 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:35.557368 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260928 05:22:35.557673 22893 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46227: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46227: connect: Connection refused (error 111)
I20260928 05:22:35.613211 22819 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41089 } attempt: 5
Sep 28 05:22:35 dist-test-slave-whsn krb5kdc[22739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572954, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260928 05:22:35.625627 22897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:35.613991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56288 (local address 127.0.97.61:41089)
0928 05:22:35.614156 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:35.614161 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:35.614182 (+ 21us) server_negotiation.cc:407] Connection header received
0928 05:22:35.614305 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:35.614310 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:35.614386 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:35.614466 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:35.615283 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:35.616005 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:35.617262 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:35.617661 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:35.617767 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:35.620908 (+ 3141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:35.620913 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:35.620916 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:22:35.620947 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:35.623289 (+ 2342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:35.623871 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:35.623876 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:35.623878 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:35.623991 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:35.624322 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:35.624326 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:35.624329 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:35.624583 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:35.624710 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:35.625123 (+ 413us) server_negotiation.cc:299] Negotiation successful
0928 05:22:35.625329 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 05:22:35.625620 22823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:35.613856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41089 (local address 127.0.0.1:56288)
0928 05:22:35.614038 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:35.614058 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 05:22:35.614179 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:22:35.614537 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:22:35.614542 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:22:35.614947 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:22:35.615128 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:35.615139 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:35.616147 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:35.616152 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:35.617122 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:35.617135 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:35.617804 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:35.617809 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:35.617989 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:35.618585 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:22:35.618603 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:22:35.620699 (+ 2096us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:22:35.623417 (+ 2718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:22:35.623423 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:22:35.623427 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:22:35.623760 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:22:35.624088 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:22:35.624093 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:22:35.624097 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:22:35.624223 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:22:35.624693 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:22:35.624697 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:22:35.624846 (+ 149us) client_negotiation.cc:769] Sending connection context
0928 05:22:35.625007 (+ 161us) client_negotiation.cc:240] Negotiation successful
0928 05:22:35.625265 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260928 05:22:35.626644 22819 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46227 } has no permanent_uuid. Determining permanent_uuid...
I20260928 05:22:35.627525 22823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:35.627175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46227 (local address 127.0.0.1:51308)
0928 05:22:35.627322 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:35.627395 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 05:22:35.628312 22823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:35.628026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46227 (local address 127.0.0.1:51310)
0928 05:22:35.628151 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:35.628224 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260928 05:22:35.628479 22819 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46227: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46227: connect: Connection refused (error 111)
I20260928 05:22:35.648875 22893 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46227 } attempt: 3
I20260928 05:22:35.649735 22896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:35.649345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46227 (local address 127.0.0.1:51314)
0928 05:22:35.649522 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:35.649597 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260928 05:22:35.650629 22896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:35.650249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46227 (local address 127.0.0.1:51328)
0928 05:22:35.650453 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:35.650520 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
W20260928 05:22:35.650802 22893 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46227: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46227: connect: Connection refused (error 111)
I20260928 05:22:35.679659 22819 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46227 } attempt: 1
I20260928 05:22:35.680588 22823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:35.680180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46227 (local address 127.0.0.1:51338)
0928 05:22:35.680348 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:35.680423 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260928 05:22:35.681586 22823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:35.681297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46227 (local address 127.0.0.1:51340)
0928 05:22:35.681428 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:35.681489 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260928 05:22:35.681761 22819 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46227: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46227: connect: Connection refused (error 111)
W20260928 05:22:35.748306 22900 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 05:22:35.748631 22900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:22:35.748721 22900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:22:35.748828 22900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260928 05:22:35.749938 22819 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46227 } attempt: 2
I20260928 05:22:35.750880 22823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:35.750466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46227 (local address 127.0.0.1:51356)
0928 05:22:35.750652 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:35.750729 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260928 05:22:35.751838 22823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:35.751543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46227 (local address 127.0.0.1:51362)
0928 05:22:35.751662 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:35.751727 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260928 05:22:35.752014 22819 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46227: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46227: connect: Connection refused (error 111)
W20260928 05:22:35.756176 22900 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:22:35.756253 22900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:22:35.756347 22900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:22:35.756404 22900 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:22:35.756484 22900 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:22:35.766309 22900 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572622576160-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572622576160-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44509,127.0.97.61:41089,127.0.97.60:46227
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572622576160-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:46227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572622576160-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572622576160-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:22:35.767524 22900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:22:35.768765 22900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:22:35.779636 22906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:22:35.779657 22908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:22:35.779592 22905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:22:35.780539 22900 server_base.cc:1034] running on GCE node
I20260928 05:22:35.781261 22900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:22:35.782567 22900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:22:35.783761 22900 hybrid_clock.cc:648] HybridClock initialized: now 1790572955783697 us; error 37 us; skew 500 ppm
I20260928 05:22:35.784053 22900 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:22:35 dist-test-slave-whsn krb5kdc[22739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572955, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:22:35.787045 22900 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260928 05:22:35.788437 22900 webserver.cc:460] Webserver started at http://127.0.97.60:37129/ using document root <none> and password file <none>
I20260928 05:22:35.789139 22900 fs_manager.cc:359] Metadata directory not provided
I20260928 05:22:35.789225 22900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:22:35.789496 22900 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:22:35.792750 22900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572622576160-388-0/minicluster-data/master-2/data/instance:
uuid: "530d59533f8641a28d9c22dc9687a544"
format_stamp: "Formatted at 2026-09-28 05:22:35 on dist-test-slave-whsn"
server_key: "305d994b9b6d1e944477af12b12e36c7"
server_key_iv: "ea710bf3854167cc24db1c14eaaae0aa"
server_key_version: "encryptionkey@0"
I20260928 05:22:35.793609 22900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572622576160-388-0/minicluster-data/master-2/wal/instance:
uuid: "530d59533f8641a28d9c22dc9687a544"
format_stamp: "Formatted at 2026-09-28 05:22:35 on dist-test-slave-whsn"
server_key: "305d994b9b6d1e944477af12b12e36c7"
server_key_iv: "ea710bf3854167cc24db1c14eaaae0aa"
server_key_version: "encryptionkey@0"
I20260928 05:22:35.793990 22893 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46227 } attempt: 4
I20260928 05:22:35.794955 22896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:35.794524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46227 (local address 127.0.0.1:51368)
0928 05:22:35.794730 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:35.794808 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260928 05:22:35.795881 22896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:35.795567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46227 (local address 127.0.0.1:51374)
0928 05:22:35.795717 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:35.795782 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260928 05:22:35.796103 22893 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46227: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46227: connect: Connection refused (error 111)
I20260928 05:22:35.799031 22900 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 05:22:35.802453 22915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:22:35.803779 22900 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 05:22:35.803925 22900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572622576160-388-0/minicluster-data/master-2/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572622576160-388-0/minicluster-data/master-2/wal
uuid: "530d59533f8641a28d9c22dc9687a544"
format_stamp: "Formatted at 2026-09-28 05:22:35 on dist-test-slave-whsn"
server_key: "305d994b9b6d1e944477af12b12e36c7"
server_key_iv: "ea710bf3854167cc24db1c14eaaae0aa"
server_key_version: "encryptionkey@0"
I20260928 05:22:35.804116 22900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572622576160-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572622576160-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572622576160-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:22:35.832679 22900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:22:35.837061 22900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:22:35.837306 22900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:22:35.854884 22900 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:46227
I20260928 05:22:35.854904 22966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:46227 every 8 connection(s)
I20260928 05:22:35.856302 22900 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572622576160-388-0/minicluster-data/master-2/data/info.pb
I20260928 05:22:35.858206 22819 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46227 } attempt: 3
I20260928 05:22:35.860226 22967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:22:35.865721 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 22900
I20260928 05:22:35.865883 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572622576160-388-0/minicluster-data/master-2/wal/instance
I20260928 05:22:35.866304 388 external_mini_cluster.cc:1428] Setting key 1a77b361b14734be6e5d85389b041ced
I20260928 05:22:35.869386 22967 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44509 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 05:22:35 dist-test-slave-whsn krb5kdc[22739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572954, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 28 05:22:35 dist-test-slave-whsn krb5kdc[22739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572954, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260928 05:22:35.880200 22823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:35.858825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46227 (local address 127.0.0.1:51390)
0928 05:22:35.859058 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:35.859078 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 05:22:35.859194 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:22:35.863461 (+ 4267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:22:35.863467 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:22:35.863830 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:22:35.863981 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:35.863991 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:35.866236 (+ 2245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:35.866242 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:35.867566 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:35.867586 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:35.869108 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:35.869114 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:35.869308 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:35.869908 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:22:35.869926 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:22:35.872143 (+ 2217us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:22:35.877254 (+ 5111us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:22:35.877269 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:22:35.877273 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:22:35.877729 (+ 456us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:22:35.878197 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:22:35.878202 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:22:35.878206 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:22:35.878369 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:22:35.879312 (+ 943us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:22:35.879318 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:22:35.879524 (+ 206us) client_negotiation.cc:769] Sending connection context
0928 05:22:35.879674 (+ 150us) client_negotiation.cc:240] Negotiation successful
0928 05:22:35.879921 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
Sep 28 05:22:35 dist-test-slave-whsn krb5kdc[22739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572955, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:22:35.882344 22969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:35.859615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51390 (local address 127.0.97.60:46227)
0928 05:22:35.860352 (+ 737us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:35.860367 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:35.860413 (+ 46us) server_negotiation.cc:407] Connection header received
0928 05:22:35.862581 (+ 2168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:35.862607 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:35.863058 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:35.863492 (+ 434us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:35.864134 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:35.866079 (+ 1945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:35.868248 (+ 2169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:35.868929 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:35.869685 (+ 756us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:35.872471 (+ 2786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:35.872482 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:35.872501 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:22:35.872540 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:35.877114 (+ 4574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:35.877844 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:35.877853 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:35.877861 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:35.878056 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:35.878475 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:35.878481 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:35.878484 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:35.879129 (+ 645us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:35.879317 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:35.880611 (+ 1294us) server_negotiation.cc:299] Negotiation successful
0928 05:22:35.881001 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":299,"threads_started":1}
I20260928 05:22:35.885550 22970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:35.867201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51398 (local address 127.0.97.60:46227)
0928 05:22:35.868190 (+ 989us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:35.868194 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:35.869401 (+ 1207us) server_negotiation.cc:407] Connection header received
0928 05:22:35.869559 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:35.869564 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:35.869639 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:35.869734 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:35.871592 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:35.872930 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:35.874641 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:35.875054 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:35.875153 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:35.879597 (+ 4444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:35.879602 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:35.879604 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:22:35.879630 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:35.882854 (+ 3224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:35.883585 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:35.883590 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:35.883594 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:35.883736 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:35.884144 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:35.884149 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:35.884151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:35.884445 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:35.884716 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:35.884962 (+ 246us) server_negotiation.cc:299] Negotiation successful
0928 05:22:35.885141 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":880,"thread_start_us":720,"threads_started":1}
I20260928 05:22:35.887401 22822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:35.872183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35480 (local address 127.0.97.62:44509)
0928 05:22:35.872363 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:35.872368 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:35.873194 (+ 826us) server_negotiation.cc:407] Connection header received
0928 05:22:35.873407 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:35.873413 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:35.873474 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:35.873548 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:35.874710 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:35.875374 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:35.876801 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:35.877294 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:35.877418 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:35.881393 (+ 3975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:35.881400 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:35.881405 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:22:35.881440 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:35.884158 (+ 2718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:35.884976 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:35.884981 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:35.884984 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:35.885135 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:35.885586 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:35.885591 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:35.885595 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:35.885989 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:35.886128 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:35.886821 (+ 693us) server_negotiation.cc:299] Negotiation successful
0928 05:22:35.887040 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 05:22:35.887410 22972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:35.872269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44509 (local address 127.0.0.1:35480)
0928 05:22:35.873061 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:35.873112 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 05:22:35.873304 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:22:35.873610 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:22:35.873616 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:22:35.874268 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:22:35.874542 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:35.874555 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:35.875536 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:35.875540 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:35.876671 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:35.876685 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:35.877474 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:35.877490 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:35.877665 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:35.878703 (+ 1038us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:22:35.878718 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:22:35.881205 (+ 2487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:22:35.884282 (+ 3077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:22:35.884291 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:22:35.884298 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:22:35.884819 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:22:35.885284 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:22:35.885288 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:22:35.885291 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:22:35.885467 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:22:35.886123 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:22:35.886130 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:22:35.886327 (+ 197us) client_negotiation.cc:769] Sending connection context
0928 05:22:35.886668 (+ 341us) client_negotiation.cc:240] Negotiation successful
0928 05:22:35.886985 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":399,"threads_started":1}
I20260928 05:22:35.889161 22967 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41089 } has no permanent_uuid. Determining permanent_uuid...
I20260928 05:22:35.890244 22819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea47b77ae9dc44a2b760efd9ed254d9b: Bootstrap starting.
I20260928 05:22:35.894322 22819 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea47b77ae9dc44a2b760efd9ed254d9b: Neither blocks nor log segments found. Creating new log.
I20260928 05:22:35.895531 22819 log.cc:824] T 00000000000000000000000000000000 P ea47b77ae9dc44a2b760efd9ed254d9b: Log is configured to *not* fsync() on all Append() calls
Sep 28 05:22:35 dist-test-slave-whsn krb5kdc[22739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572955, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Sep 28 05:22:35 dist-test-slave-whsn krb5kdc[22739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572954, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:22:35.898537 22819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea47b77ae9dc44a2b760efd9ed254d9b: No bootstrap required, opened a new log
I20260928 05:22:35.903900 22972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:35.889959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41089 (local address 127.0.0.1:56298)
0928 05:22:35.890143 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:35.890162 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 05:22:35.890265 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:22:35.890746 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:22:35.890750 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:22:35.891091 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:22:35.891241 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:35.891249 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:35.892405 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:35.892411 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:35.893789 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:35.893805 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:35.894741 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:35.894747 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:35.894912 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:35.895468 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:22:35.895490 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:22:35.898085 (+ 2595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:22:35.901510 (+ 3425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:22:35.901515 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:22:35.901519 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:22:35.902005 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:22:35.902438 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:22:35.902443 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:22:35.902446 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:22:35.902571 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:22:35.903038 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:22:35.903042 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:22:35.903183 (+ 141us) client_negotiation.cc:769] Sending connection context
0928 05:22:35.903301 (+ 118us) client_negotiation.cc:240] Negotiation successful
0928 05:22:35.903547 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260928 05:22:35.903924 22897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:35.890329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56298 (local address 127.0.97.61:41089)
0928 05:22:35.890503 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:35.890507 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:35.890521 (+ 14us) server_negotiation.cc:407] Connection header received
0928 05:22:35.890564 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:35.890568 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:35.890620 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:35.890709 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:35.891378 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:35.892250 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:35.893941 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:35.894435 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:35.894548 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:35.898258 (+ 3710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:35.898264 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:35.898268 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:22:35.898297 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:35.901363 (+ 3066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:35.902139 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:35.902145 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:35.902149 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:35.902304 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:35.902670 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:35.902675 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:35.902678 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:35.902920 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:35.903057 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:35.903402 (+ 345us) server_negotiation.cc:299] Negotiation successful
0928 05:22:35.903593 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 05:22:35.905357 22822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:35.887617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35488 (local address 127.0.97.62:44509)
0928 05:22:35.888007 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:35.888012 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:35.888480 (+ 468us) server_negotiation.cc:407] Connection header received
0928 05:22:35.888690 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:35.888696 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:35.888770 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:35.888906 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:35.890420 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:35.891620 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:35.893043 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:35.893528 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:35.893641 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:35.898788 (+ 5147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:35.898793 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:35.898796 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:22:35.898822 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:35.902275 (+ 3453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:35.903114 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:35.903120 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:35.903123 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:35.903291 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:35.903753 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:35.903758 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:35.903762 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:35.904150 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:35.904438 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:35.904702 (+ 264us) server_negotiation.cc:299] Negotiation successful
0928 05:22:35.904921 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":591,"threads_started":1}
I20260928 05:22:35.905468 22967 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46227 } has no permanent_uuid. Determining permanent_uuid...
I20260928 05:22:35.909164 22819 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea47b77ae9dc44a2b760efd9ed254d9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea47b77ae9dc44a2b760efd9ed254d9b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44509 } } peers { permanent_uuid: "7224764aaf5c4cfeb98f8dce2b2e47fb" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41089 } } peers { permanent_uuid: "530d59533f8641a28d9c22dc9687a544" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46227 } }
I20260928 05:22:35.909593 22819 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea47b77ae9dc44a2b760efd9ed254d9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea47b77ae9dc44a2b760efd9ed254d9b, State: Initialized, Role: FOLLOWER
I20260928 05:22:35.910131 22819 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea47b77ae9dc44a2b760efd9ed254d9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea47b77ae9dc44a2b760efd9ed254d9b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44509 } } peers { permanent_uuid: "7224764aaf5c4cfeb98f8dce2b2e47fb" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41089 } } peers { permanent_uuid: "530d59533f8641a28d9c22dc9687a544" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46227 } }
I20260928 05:22:35.911139 22975 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea47b77ae9dc44a2b760efd9ed254d9b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea47b77ae9dc44a2b760efd9ed254d9b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44509 } } peers { permanent_uuid: "7224764aaf5c4cfeb98f8dce2b2e47fb" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41089 } } peers { permanent_uuid: "530d59533f8641a28d9c22dc9687a544" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46227 } } }
I20260928 05:22:35.911684 22975 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea47b77ae9dc44a2b760efd9ed254d9b [sys.catalog]: This master's current role is: FOLLOWER
I20260928 05:22:35.912387 22819 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea47b77ae9dc44a2b760efd9ed254d9b [sys.catalog]: configured and running, proceeding with master startup.
Sep 28 05:22:35 dist-test-slave-whsn krb5kdc[22739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572954, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260928 05:22:35.924136 22897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:35.907151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56308 (local address 127.0.97.61:41089)
0928 05:22:35.907292 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:35.907296 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:35.907312 (+ 16us) server_negotiation.cc:407] Connection header received
0928 05:22:35.907418 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:35.907423 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:35.907475 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:35.907546 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:35.909004 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:35.910144 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:35.911789 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:35.912394 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:35.912485 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:35.917874 (+ 5389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:35.917880 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:35.917885 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:22:35.917921 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:35.921252 (+ 3331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:35.921975 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:35.921981 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:35.921985 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:35.922137 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:35.922826 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:35.922831 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:35.922835 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:35.923076 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:35.923323 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:35.923612 (+ 289us) server_negotiation.cc:299] Negotiation successful
0928 05:22:35.923844 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
Sep 28 05:22:35 dist-test-slave-whsn krb5kdc[22739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572955, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260928 05:22:35.928879 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 05:22:35.932025 22794 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "ea47b77ae9dc44a2b760efd9ed254d9b"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35488
I20260928 05:22:35.932441 22972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:35.906564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46227 (local address 127.0.0.1:51408)
0928 05:22:35.906740 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:35.906758 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 05:22:35.906843 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:22:35.907456 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:22:35.907461 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:22:35.907957 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:22:35.908181 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:35.908194 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:35.919176 (+ 10982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:35.919183 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:35.920683 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:35.920702 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:35.921820 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:35.921827 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:35.922035 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:35.922596 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:22:35.922635 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:22:35.925464 (+ 2829us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:22:35.929559 (+ 4095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:22:35.929564 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:22:35.929569 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:22:35.930078 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:22:35.930510 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:22:35.930514 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:22:35.930519 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:22:35.930666 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:22:35.931341 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:22:35.931347 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:22:35.931498 (+ 151us) client_negotiation.cc:769] Sending connection context
0928 05:22:35.931664 (+ 166us) client_negotiation.cc:240] Negotiation successful
0928 05:22:35.931992 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260928 05:22:35.932436 22794 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea47b77ae9dc44a2b760efd9ed254d9b [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 05:22:35.932628 22794 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea47b77ae9dc44a2b760efd9ed254d9b [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:22:35.933619 22970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:35.906872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51408 (local address 127.0.97.60:46227)
0928 05:22:35.907041 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:35.907047 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:35.907067 (+ 20us) server_negotiation.cc:407] Connection header received
0928 05:22:35.907128 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:35.907133 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:35.907201 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:35.907301 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:35.908358 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:35.919010 (+ 10652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:35.920919 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:35.921644 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:35.922339 (+ 695us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:35.925625 (+ 3286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:35.925631 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:35.925636 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:22:35.925669 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:35.929410 (+ 3741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:35.930247 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:35.930251 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:35.930255 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:35.930395 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:35.930797 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:35.930803 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:35.930807 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:35.931206 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:35.932816 (+ 1610us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:35.932998 (+ 182us) server_negotiation.cc:299] Negotiation successful
0928 05:22:35.933241 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 05:22:35.936205 22794 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea47b77ae9dc44a2b760efd9ed254d9b [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea47b77ae9dc44a2b760efd9ed254d9b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44509 } } peers { permanent_uuid: "7224764aaf5c4cfeb98f8dce2b2e47fb" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41089 } } peers { permanent_uuid: "530d59533f8641a28d9c22dc9687a544" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46227 } }
I20260928 05:22:35.937814 22794 leader_election.cc:288] T 00000000000000000000000000000000 P ea47b77ae9dc44a2b760efd9ed254d9b [CANDIDATE]: Term 1 election: Requested vote from peers 7224764aaf5c4cfeb98f8dce2b2e47fb (127.0.97.61:41089), 530d59533f8641a28d9c22dc9687a544 (127.0.97.60:46227)
W20260928 05:22:35.942412 22986 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ea47b77ae9dc44a2b760efd9ed254d9b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 05:22:35.942546 22986 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 05:22:35.942708 22967 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 530d59533f8641a28d9c22dc9687a544: Bootstrap starting.
I20260928 05:22:35.930262 22752 master_runner.cc:186] Error getting master registration for 127.0.97.61:41089: OK, instance_id { permanent_uuid: "7224764aaf5c4cfeb98f8dce2b2e47fb" instance_seqno: 1790572955375896 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 05:22:35.946269 22752 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 05:22:35.946434 22967 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 530d59533f8641a28d9c22dc9687a544: Neither blocks nor log segments found. Creating new log.
I20260928 05:22:35.947643 22967 log.cc:824] T 00000000000000000000000000000000 P 530d59533f8641a28d9c22dc9687a544: Log is configured to *not* fsync() on all Append() calls
I20260928 05:22:35.950897 22967 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 530d59533f8641a28d9c22dc9687a544: No bootstrap required, opened a new log
I20260928 05:22:35.961247 22967 raft_consensus.cc:348] T 00000000000000000000000000000000 P 530d59533f8641a28d9c22dc9687a544 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea47b77ae9dc44a2b760efd9ed254d9b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44509 } } peers { permanent_uuid: "7224764aaf5c4cfeb98f8dce2b2e47fb" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41089 } } peers { permanent_uuid: "530d59533f8641a28d9c22dc9687a544" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46227 } }
I20260928 05:22:35.961711 22967 raft_consensus.cc:729] T 00000000000000000000000000000000 P 530d59533f8641a28d9c22dc9687a544 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 530d59533f8641a28d9c22dc9687a544, State: Initialized, Role: FOLLOWER
I20260928 05:22:35.962201 22967 consensus_queue.cc:260] T 00000000000000000000000000000000 P 530d59533f8641a28d9c22dc9687a544 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea47b77ae9dc44a2b760efd9ed254d9b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44509 } } peers { permanent_uuid: "7224764aaf5c4cfeb98f8dce2b2e47fb" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41089 } } peers { permanent_uuid: "530d59533f8641a28d9c22dc9687a544" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46227 } }
I20260928 05:22:35.969049 22996 sys_catalog.cc:455] T 00000000000000000000000000000000 P 530d59533f8641a28d9c22dc9687a544 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea47b77ae9dc44a2b760efd9ed254d9b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44509 } } peers { permanent_uuid: "7224764aaf5c4cfeb98f8dce2b2e47fb" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41089 } } peers { permanent_uuid: "530d59533f8641a28d9c22dc9687a544" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46227 } } }
I20260928 05:22:35.969400 22996 sys_catalog.cc:458] T 00000000000000000000000000000000 P 530d59533f8641a28d9c22dc9687a544 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 05:22:35.971132 22967 sys_catalog.cc:564] T 00000000000000000000000000000000 P 530d59533f8641a28d9c22dc9687a544 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:22:35.971191 22942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ea47b77ae9dc44a2b760efd9ed254d9b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "530d59533f8641a28d9c22dc9687a544"
I20260928 05:22:35.971737 22942 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 530d59533f8641a28d9c22dc9687a544 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:22:35.975421 22942 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 530d59533f8641a28d9c22dc9687a544 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea47b77ae9dc44a2b760efd9ed254d9b in term 1.
I20260928 05:22:35.976202 22768 leader_election.cc:302] T 00000000000000000000000000000000 P ea47b77ae9dc44a2b760efd9ed254d9b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 530d59533f8641a28d9c22dc9687a544, ea47b77ae9dc44a2b760efd9ed254d9b; no voters:
I20260928 05:22:35.976780 22975 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea47b77ae9dc44a2b760efd9ed254d9b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:22:35.977481 22975 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea47b77ae9dc44a2b760efd9ed254d9b [term 1 LEADER]: Becoming Leader. State: Replica: ea47b77ae9dc44a2b760efd9ed254d9b, State: Running, Role: LEADER
I20260928 05:22:35.978044 22975 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea47b77ae9dc44a2b760efd9ed254d9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea47b77ae9dc44a2b760efd9ed254d9b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44509 } } peers { permanent_uuid: "7224764aaf5c4cfeb98f8dce2b2e47fb" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41089 } } peers { permanent_uuid: "530d59533f8641a28d9c22dc9687a544" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46227 } }
I20260928 05:22:35.981462 22998 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea47b77ae9dc44a2b760efd9ed254d9b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea47b77ae9dc44a2b760efd9ed254d9b. Latest consensus state: current_term: 1 leader_uuid: "ea47b77ae9dc44a2b760efd9ed254d9b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea47b77ae9dc44a2b760efd9ed254d9b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44509 } } peers { permanent_uuid: "7224764aaf5c4cfeb98f8dce2b2e47fb" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41089 } } peers { permanent_uuid: "530d59533f8641a28d9c22dc9687a544" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46227 } } }
I20260928 05:22:35.981923 22998 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea47b77ae9dc44a2b760efd9ed254d9b [sys.catalog]: This master's current role is: LEADER
I20260928 05:22:35.993675 23002 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:22:35.996078 23002 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:22:35.996356 22897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:35.960165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56324 (local address 127.0.97.61:41089)
0928 05:22:35.960346 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:35.960354 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:35.961296 (+ 942us) server_negotiation.cc:407] Connection header received
0928 05:22:35.961819 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:35.961825 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:35.961909 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:35.961984 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:35.965786 (+ 3802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:35.966967 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:35.970446 (+ 3479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:35.971026 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:35.971122 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:35.984789 (+ 13667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:35.984798 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:35.984804 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:22:35.984870 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:35.989021 (+ 4151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:35.991722 (+ 2701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:35.991729 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:35.991732 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:35.991914 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:35.993164 (+ 1250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:35.993171 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:35.993175 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:35.993584 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:35.993838 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:35.995402 (+ 1564us) server_negotiation.cc:299] Negotiation successful
0928 05:22:35.995783 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 05:22:36.003814 22822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:35.959661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35500 (local address 127.0.97.62:44509)
0928 05:22:35.959840 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:35.959846 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:35.962314 (+ 2468us) server_negotiation.cc:407] Connection header received
0928 05:22:35.962762 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:35.962769 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:35.962861 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:35.962940 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:35.967058 (+ 4118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:35.968175 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:35.976081 (+ 7906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:35.976879 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:35.977007 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:35.996170 (+ 19163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:35.996179 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:35.996183 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:22:35.996227 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:35.999935 (+ 3708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:36.000964 (+ 1029us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:36.000971 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:36.000976 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:36.001209 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:36.001706 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:36.001712 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:36.001716 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:36.002106 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:36.002394 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:36.003070 (+ 676us) server_negotiation.cc:299] Negotiation successful
0928 05:22:36.003342 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 05:22:36.013059 22942 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 530d59533f8641a28d9c22dc9687a544 [term 1 FOLLOWER]: Refusing update from remote peer ea47b77ae9dc44a2b760efd9ed254d9b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:22:36.014154 22998 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ea47b77ae9dc44a2b760efd9ed254d9b [LEADER]: Connected to new peer: Peer: permanent_uuid: "530d59533f8641a28d9c22dc9687a544" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:22:36.016024 22970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:35.960920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51416 (local address 127.0.97.60:46227)
0928 05:22:35.961121 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:35.961128 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:35.974205 (+ 13077us) server_negotiation.cc:407] Connection header received
0928 05:22:35.974910 (+ 705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:35.974918 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:35.975011 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:35.975086 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:35.982230 (+ 7144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:35.983411 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:35.997578 (+ 14167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:35.998356 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:35.998463 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:36.006846 (+ 8383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:36.006854 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:36.006879 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:22:36.006926 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:36.010790 (+ 3864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:36.012071 (+ 1281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:36.012078 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:36.012082 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:36.012260 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:36.013075 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:36.013081 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:36.013085 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:36.013671 (+ 586us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:36.014061 (+ 390us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:36.014984 (+ 923us) server_negotiation.cc:299] Negotiation successful
0928 05:22:36.015388 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 05:22:36.024899 22996 sys_catalog.cc:455] T 00000000000000000000000000000000 P 530d59533f8641a28d9c22dc9687a544 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea47b77ae9dc44a2b760efd9ed254d9b. Latest consensus state: current_term: 1 leader_uuid: "ea47b77ae9dc44a2b760efd9ed254d9b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea47b77ae9dc44a2b760efd9ed254d9b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44509 } } peers { permanent_uuid: "7224764aaf5c4cfeb98f8dce2b2e47fb" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41089 } } peers { permanent_uuid: "530d59533f8641a28d9c22dc9687a544" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46227 } } }
I20260928 05:22:36.025243 22996 sys_catalog.cc:458] T 00000000000000000000000000000000 P 530d59533f8641a28d9c22dc9687a544 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 05:22:36.029021 22998 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea47b77ae9dc44a2b760efd9ed254d9b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ea47b77ae9dc44a2b760efd9ed254d9b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea47b77ae9dc44a2b760efd9ed254d9b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44509 } } peers { permanent_uuid: "7224764aaf5c4cfeb98f8dce2b2e47fb" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41089 } } peers { permanent_uuid: "530d59533f8641a28d9c22dc9687a544" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46227 } } }
I20260928 05:22:36.029399 22998 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea47b77ae9dc44a2b760efd9ed254d9b [sys.catalog]: This master's current role is: LEADER
I20260928 05:22:36.030599 23009 mvcc.cc:205] Tried to move back new op lower bound from 7334186827816185856 to 7334186827691102208. Current Snapshot: MvccSnapshot[applied={T|T < 7334186827816185856}]
I20260928 05:22:36.034961 23002 catalog_manager.cc:1347] Generated new cluster ID: 3bd607c4d00446fa88d91206efc37dd5
I20260928 05:22:36.035077 23002 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:22:36.049228 22996 sys_catalog.cc:455] T 00000000000000000000000000000000 P 530d59533f8641a28d9c22dc9687a544 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ea47b77ae9dc44a2b760efd9ed254d9b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea47b77ae9dc44a2b760efd9ed254d9b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44509 } } peers { permanent_uuid: "7224764aaf5c4cfeb98f8dce2b2e47fb" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41089 } } peers { permanent_uuid: "530d59533f8641a28d9c22dc9687a544" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46227 } } }
I20260928 05:22:36.049503 22996 sys_catalog.cc:458] T 00000000000000000000000000000000 P 530d59533f8641a28d9c22dc9687a544 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 05:22:36.073303 23015 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 530d59533f8641a28d9c22dc9687a544: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 05:22:36.073417 23015 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 05:22:36.061311 22900 master_runner.cc:186] Error getting master registration for 127.0.97.61:41089: OK, instance_id { permanent_uuid: "7224764aaf5c4cfeb98f8dce2b2e47fb" instance_seqno: 1790572955375896 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 05:22:36.074388 22900 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 05:22:36.089375 23002 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:22:36.091300 23002 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:22:36.095268 22893 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46227 } attempt: 5
I20260928 05:22:36.104887 23002 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea47b77ae9dc44a2b760efd9ed254d9b: Generated new TSK 0
I20260928 05:22:36.106043 23002 catalog_manager.cc:1514] Initializing in-progress tserver states...
Sep 28 05:22:36 dist-test-slave-whsn krb5kdc[22739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572955, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260928 05:22:36.115231 22970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:36.096483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51420 (local address 127.0.97.60:46227)
0928 05:22:36.096655 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:36.096661 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:36.096677 (+ 16us) server_negotiation.cc:407] Connection header received
0928 05:22:36.096727 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:36.096732 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:36.096801 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:36.096916 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:36.098158 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:36.099347 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:36.101074 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:36.101844 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:36.101965 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:36.108136 (+ 6171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:36.108143 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:36.108148 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:22:36.108185 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:36.112055 (+ 3870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:36.112920 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:36.112926 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:36.112929 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:36.113092 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:36.113546 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:36.113552 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:36.113555 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:36.114007 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:36.114165 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:36.114633 (+ 468us) server_negotiation.cc:299] Negotiation successful
0928 05:22:36.114843 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 05:22:36.115236 22896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:36.095808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46227 (local address 127.0.0.1:51420)
0928 05:22:36.096145 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:36.096166 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 05:22:36.096295 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:22:36.096947 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:22:36.096952 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:22:36.097485 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:22:36.097681 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:36.097694 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:36.099471 (+ 1777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:36.099477 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:36.100895 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:36.100910 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:36.102967 (+ 2057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:36.102974 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:36.103181 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:36.105201 (+ 2020us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:22:36.105224 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:22:36.107959 (+ 2735us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:22:36.112246 (+ 4287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:22:36.112253 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:22:36.112257 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:22:36.112774 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:22:36.113222 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:22:36.113227 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:22:36.113231 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:22:36.113410 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:22:36.114155 (+ 745us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:22:36.114161 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:22:36.114341 (+ 180us) client_negotiation.cc:769] Sending connection context
0928 05:22:36.114502 (+ 161us) client_negotiation.cc:240] Negotiation successful
0928 05:22:36.114787 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217}
I20260928 05:22:36.121287 22893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7224764aaf5c4cfeb98f8dce2b2e47fb: Bootstrap starting.
I20260928 05:22:36.127712 22893 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7224764aaf5c4cfeb98f8dce2b2e47fb: Neither blocks nor log segments found. Creating new log.
I20260928 05:22:36.129711 22893 log.cc:824] T 00000000000000000000000000000000 P 7224764aaf5c4cfeb98f8dce2b2e47fb: Log is configured to *not* fsync() on all Append() calls
I20260928 05:22:36.150296 22893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7224764aaf5c4cfeb98f8dce2b2e47fb: No bootstrap required, opened a new log
I20260928 05:22:36.166623 22893 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7224764aaf5c4cfeb98f8dce2b2e47fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea47b77ae9dc44a2b760efd9ed254d9b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44509 } } peers { permanent_uuid: "7224764aaf5c4cfeb98f8dce2b2e47fb" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41089 } } peers { permanent_uuid: "530d59533f8641a28d9c22dc9687a544" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46227 } }
I20260928 05:22:36.167348 22893 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7224764aaf5c4cfeb98f8dce2b2e47fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7224764aaf5c4cfeb98f8dce2b2e47fb, State: Initialized, Role: FOLLOWER
I20260928 05:22:36.168180 22893 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7224764aaf5c4cfeb98f8dce2b2e47fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea47b77ae9dc44a2b760efd9ed254d9b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44509 } } peers { permanent_uuid: "7224764aaf5c4cfeb98f8dce2b2e47fb" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41089 } } peers { permanent_uuid: "530d59533f8641a28d9c22dc9687a544" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46227 } }
I20260928 05:22:36.178884 22868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ea47b77ae9dc44a2b760efd9ed254d9b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7224764aaf5c4cfeb98f8dce2b2e47fb"
I20260928 05:22:36.179344 22867 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7224764aaf5c4cfeb98f8dce2b2e47fb [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:22:36.182801 22867 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7224764aaf5c4cfeb98f8dce2b2e47fb [term 1 FOLLOWER]: Refusing update from remote peer ea47b77ae9dc44a2b760efd9ed254d9b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:22:36.185905 22868 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7224764aaf5c4cfeb98f8dce2b2e47fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea47b77ae9dc44a2b760efd9ed254d9b in term 1.
I20260928 05:22:36.188308 22998 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ea47b77ae9dc44a2b760efd9ed254d9b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7224764aaf5c4cfeb98f8dce2b2e47fb" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:22:36.191392 22893 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7224764aaf5c4cfeb98f8dce2b2e47fb [sys.catalog]: configured and running, proceeding with master startup.
W20260928 05:22:36.241647 22897 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:22:36.220447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56330 (local address 127.0.97.61:41089)
0928 05:22:36.220635 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:36.220642 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:36.232681 (+ 12039us) server_negotiation.cc:407] Connection header received
0928 05:22:36.235254 (+ 2573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:36.235262 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:36.235366 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:36.235446 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:36.239628 (+ 4182us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56330: BlockingRecv error: recv got EOF from 127.0.0.1:56330 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260928 05:22:36.244217 22822 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:22:36.228707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35512 (local address 127.0.97.62:44509)
0928 05:22:36.228949 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:36.228956 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:36.237356 (+ 8400us) server_negotiation.cc:407] Connection header received
0928 05:22:36.242478 (+ 5122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:36.242486 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:36.242578 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:36.242669 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:36.243974 (+ 1305us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35512: BlockingRecv error: recv got EOF from 127.0.0.1:35512 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260928 05:22:36.247205 22970 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:22:36.233244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51434 (local address 127.0.97.60:46227)
0928 05:22:36.233428 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:36.233434 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:36.242352 (+ 8918us) server_negotiation.cc:407] Connection header received
0928 05:22:36.245369 (+ 3017us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:36.245376 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:36.245603 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:36.245734 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:36.246937 (+ 1203us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51434: BlockingRecv error: recv got EOF from 127.0.0.1:51434 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 05:22:36.258481 23033 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7224764aaf5c4cfeb98f8dce2b2e47fb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ea47b77ae9dc44a2b760efd9ed254d9b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea47b77ae9dc44a2b760efd9ed254d9b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44509 } } peers { permanent_uuid: "7224764aaf5c4cfeb98f8dce2b2e47fb" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41089 } } peers { permanent_uuid: "530d59533f8641a28d9c22dc9687a544" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46227 } } }
I20260928 05:22:36.258967 23033 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7224764aaf5c4cfeb98f8dce2b2e47fb [sys.catalog]: This master's current role is: FOLLOWER
I20260928 05:22:36.259336 23019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7224764aaf5c4cfeb98f8dce2b2e47fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea47b77ae9dc44a2b760efd9ed254d9b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea47b77ae9dc44a2b760efd9ed254d9b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44509 } } peers { permanent_uuid: "7224764aaf5c4cfeb98f8dce2b2e47fb" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41089 } } peers { permanent_uuid: "530d59533f8641a28d9c22dc9687a544" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46227 } } }
I20260928 05:22:36.259686 23019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7224764aaf5c4cfeb98f8dce2b2e47fb [sys.catalog]: This master's current role is: FOLLOWER
I20260928 05:22:36.259982 23022 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7224764aaf5c4cfeb98f8dce2b2e47fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea47b77ae9dc44a2b760efd9ed254d9b. Latest consensus state: current_term: 1 leader_uuid: "ea47b77ae9dc44a2b760efd9ed254d9b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea47b77ae9dc44a2b760efd9ed254d9b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44509 } } peers { permanent_uuid: "7224764aaf5c4cfeb98f8dce2b2e47fb" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41089 } } peers { permanent_uuid: "530d59533f8641a28d9c22dc9687a544" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46227 } } }
I20260928 05:22:36.260291 23022 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7224764aaf5c4cfeb98f8dce2b2e47fb [sys.catalog]: This master's current role is: FOLLOWER
I20260928 05:22:36.285188 23048 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260928 05:22:36.298982 22897 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:22:36.293352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56338 (local address 127.0.97.61:41089)
0928 05:22:36.293535 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:36.293543 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:36.297420 (+ 3877us) server_negotiation.cc:407] Connection header received
0928 05:22:36.297616 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:36.297623 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:36.297754 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:36.297871 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:36.298703 (+ 832us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56338: BlockingRecv error: recv got EOF from 127.0.0.1:56338 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260928 05:22:36.314105 22897 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:22:36.305124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56348 (local address 127.0.97.61:41089)
0928 05:22:36.305280 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:36.305286 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:36.312554 (+ 7268us) server_negotiation.cc:407] Connection header received
0928 05:22:36.312789 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:36.312796 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:36.312914 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:36.313015 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:36.313890 (+ 875us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56348: BlockingRecv error: recv got EOF from 127.0.0.1:56348 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260928 05:22:36.322882 22897 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:22:36.319912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56362 (local address 127.0.97.61:41089)
0928 05:22:36.320103 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:36.320109 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:36.321409 (+ 1300us) server_negotiation.cc:407] Connection header received
0928 05:22:36.321641 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:36.321647 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:36.321747 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:36.321849 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:36.322641 (+ 792us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56362: BlockingRecv error: recv got EOF from 127.0.0.1:56362 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260928 05:22:36.332680 22897 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:22:36.330625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56378 (local address 127.0.97.61:41089)
0928 05:22:36.330834 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:36.330841 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:36.331319 (+ 478us) server_negotiation.cc:407] Connection header received
0928 05:22:36.331493 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:36.331498 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:36.331588 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:36.331707 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:36.332437 (+ 730us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56378: BlockingRecv error: recv got EOF from 127.0.0.1:56378 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
W20260928 05:22:36.346232 22897 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:22:36.343975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56386 (local address 127.0.97.61:41089)
0928 05:22:36.344137 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:36.344143 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:36.344652 (+ 509us) server_negotiation.cc:407] Connection header received
0928 05:22:36.344840 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:36.344846 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:36.344965 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:36.345069 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:36.346004 (+ 935us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56386: BlockingRecv error: recv got EOF from 127.0.0.1:56386 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260928 05:22:36.367950 22897 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:22:36.365822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56388 (local address 127.0.97.61:41089)
0928 05:22:36.365990 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:36.365996 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:36.366470 (+ 474us) server_negotiation.cc:407] Connection header received
0928 05:22:36.366663 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:36.366669 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:36.366787 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:36.366929 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:36.367716 (+ 787us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56388: BlockingRecv error: recv got EOF from 127.0.0.1:56388 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260928 05:22:36.411232 22897 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:22:36.409012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56398 (local address 127.0.97.61:41089)
0928 05:22:36.409193 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:36.409199 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:36.409754 (+ 555us) server_negotiation.cc:407] Connection header received
0928 05:22:36.409947 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:36.409954 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:36.410050 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:36.410165 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:36.410994 (+ 829us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56398: BlockingRecv error: recv got EOF from 127.0.0.1:56398 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260928 05:22:36.480679 22897 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:22:36.478830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56408 (local address 127.0.97.61:41089)
0928 05:22:36.479067 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:36.479073 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:36.479324 (+ 251us) server_negotiation.cc:407] Connection header received
0928 05:22:36.479540 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:36.479544 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:36.479615 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:36.479694 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:36.480464 (+ 770us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56408: BlockingRecv error: recv got EOF from 127.0.0.1:56408 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
I20260928 05:22:36.578356 22998 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea47b77ae9dc44a2b760efd9ed254d9b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ea47b77ae9dc44a2b760efd9ed254d9b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea47b77ae9dc44a2b760efd9ed254d9b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44509 } } peers { permanent_uuid: "7224764aaf5c4cfeb98f8dce2b2e47fb" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41089 } } peers { permanent_uuid: "530d59533f8641a28d9c22dc9687a544" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46227 } } }
I20260928 05:22:36.578663 22998 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea47b77ae9dc44a2b760efd9ed254d9b [sys.catalog]: This master's current role is: LEADER
W20260928 05:22:36.613957 22897 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:22:36.612112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56424 (local address 127.0.97.61:41089)
0928 05:22:36.612327 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:36.612334 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:36.612580 (+ 246us) server_negotiation.cc:407] Connection header received
0928 05:22:36.612790 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:36.612794 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:36.612894 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:36.612988 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:36.613697 (+ 709us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56424: BlockingRecv error: recv got EOF from 127.0.0.1:56424 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260928 05:22:36.875031 22897 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:22:36.873338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56436 (local address 127.0.97.61:41089)
0928 05:22:36.873540 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:36.873547 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:36.873765 (+ 218us) server_negotiation.cc:407] Connection header received
0928 05:22:36.873968 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:36.873973 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:36.874041 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:36.874124 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:36.874831 (+ 707us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56436: BlockingRecv error: recv got EOF from 127.0.0.1:56436 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 05:22:37.075027 23015 catalog_manager.cc:1259] Loaded cluster ID: 3bd607c4d00446fa88d91206efc37dd5
I20260928 05:22:37.075129 23015 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 530d59533f8641a28d9c22dc9687a544: loading cluster ID for follower catalog manager: success
I20260928 05:22:37.077625 23015 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 530d59533f8641a28d9c22dc9687a544: acquiring CA information for follower catalog manager: success
I20260928 05:22:37.079593 23015 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 530d59533f8641a28d9c22dc9687a544: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 05:22:37.289465 23048 catalog_manager.cc:1259] Loaded cluster ID: 3bd607c4d00446fa88d91206efc37dd5
I20260928 05:22:37.289582 23048 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7224764aaf5c4cfeb98f8dce2b2e47fb: loading cluster ID for follower catalog manager: success
I20260928 05:22:37.292164 23048 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7224764aaf5c4cfeb98f8dce2b2e47fb: acquiring CA information for follower catalog manager: success
I20260928 05:22:37.293903 23048 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7224764aaf5c4cfeb98f8dce2b2e47fb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 05:22:37.397459 23102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:37.390639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56442 (local address 127.0.97.61:41089)
0928 05:22:37.391239 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:37.391245 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:37.391477 (+ 232us) server_negotiation.cc:407] Connection header received
0928 05:22:37.391709 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:37.391714 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:37.391888 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:37.391999 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:22:37.392456 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:37.393304 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:37.395210 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:37.395618 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:37.395710 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:37.396257 (+ 547us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:22:37.396833 (+ 576us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:22:37.396991 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:37.397092 (+ 101us) server_negotiation.cc:299] Negotiation successful
0928 05:22:37.397157 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":266,"threads_started":1}
I20260928 05:22:37.409655 23108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:37.403600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51446 (local address 127.0.97.60:46227)
0928 05:22:37.404132 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:37.404136 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:37.404152 (+ 16us) server_negotiation.cc:407] Connection header received
0928 05:22:37.404243 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:37.404246 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:37.404391 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:37.404518 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:22:37.404937 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:37.405640 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:37.407472 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:37.407907 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:37.407996 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:37.408536 (+ 540us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:22:37.409086 (+ 550us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:22:37.409200 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:37.409306 (+ 106us) server_negotiation.cc:299] Negotiation successful
0928 05:22:37.409373 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":331,"threads_started":1}
I20260928 05:22:37.433336 23102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:37.415857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56454 (local address 127.0.97.61:41089)
0928 05:22:37.416044 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:37.416051 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:37.426458 (+ 10407us) server_negotiation.cc:407] Connection header received
0928 05:22:37.426735 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:37.426742 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:37.426824 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:37.426909 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:22:37.427629 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:37.428814 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:37.431339 (+ 2525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:37.432009 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:37.432097 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:37.432640 (+ 543us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:22:37.432737 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:22:37.432811 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:37.432970 (+ 159us) server_negotiation.cc:299] Negotiation successful
0928 05:22:37.433050 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 05:22:37.437783 23115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:37.415569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35518 (local address 127.0.97.62:44509)
0928 05:22:37.426025 (+ 10456us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:37.426031 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:37.430421 (+ 4390us) server_negotiation.cc:407] Connection header received
0928 05:22:37.430824 (+ 403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:37.430830 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:37.431030 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:37.431181 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:22:37.432571 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:37.433698 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:37.435611 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:37.436132 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:37.436243 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:37.436742 (+ 499us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:22:37.437125 (+ 383us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:22:37.437217 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:37.437329 (+ 112us) server_negotiation.cc:299] Negotiation successful
0928 05:22:37.437482 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10339,"thread_start_us":10230,"threads_started":1}
I20260928 05:22:37.441143 23108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:37.426617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51450 (local address 127.0.97.60:46227)
0928 05:22:37.426787 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:37.426792 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:37.432375 (+ 5583us) server_negotiation.cc:407] Connection header received
0928 05:22:37.435703 (+ 3328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:37.435709 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:37.435789 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:37.435888 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:22:37.436377 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:37.437485 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:37.439451 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:37.439909 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:37.440030 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:37.440464 (+ 434us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:22:37.440579 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:22:37.440651 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:37.440779 (+ 128us) server_negotiation.cc:299] Negotiation successful
0928 05:22:37.440885 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 05:22:37.441951 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 22752
I20260928 05:22:37.442864 22814 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:22:37.551100 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 22752
W20260928 05:22:37.567836 22918 connection.cc:535] server connection from 127.0.0.1:51390 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 05:22:37.568030 22842 connection.cc:535] server connection from 127.0.0.1:56288 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 05:22:37.568460 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 22826
I20260928 05:22:37.569628 22888 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:22:37.680218 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 22826
I20260928 05:22:37.695925 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 22900
I20260928 05:22:37.696981 22962 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:22:37.800750 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 22900
2026-09-28T05:22:37Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4519 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572622576160-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:22:37 dist-test-slave-whsn krb5kdc[23123](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:22:37 dist-test-slave-whsn krb5kdc[23123](info): set up 2 sockets
Sep 28 05:22:37 dist-test-slave-whsn krb5kdc[23123](info): commencing operation
krb5kdc: starting...
W20260928 05:22:38.817080 388 mini_kdc.cc:121] Time spent starting KDC: real 0.981s user 0.002s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572622576160-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572622576160-388-0/krb5kdc/test-user.keytab.
Sep 28 05:22:38 dist-test-slave-whsn krb5kdc[23123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:22:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:22:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 05:22:39.032356 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572622576160-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572622576160-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572622576160-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40853
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40853,127.0.97.61:40335,127.0.97.60:45935
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:22:39.338261 23136 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 05:22:39.338570 23136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:22:39.338671 23136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:22:39.338780 23136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:22:39.346060 23136 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:22:39.346141 23136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:22:39.346226 23136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:22:39.346287 23136 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:22:39.346369 23136 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:22:39.355595 23136 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572622576160-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572622576160-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40853,127.0.97.61:40335,127.0.97.60:45935
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572622576160-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572622576160-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:22:39.356737 23136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:22:39.357939 23136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:22:39.368636 23141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:22:39.368661 23142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:22:39.368587 23144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:22:39.369478 23136 server_base.cc:1034] running on GCE node
I20260928 05:22:39.370108 23136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:22:39.371296 23136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:22:39.372499 23136 hybrid_clock.cc:648] HybridClock initialized: now 1790572959372434 us; error 39 us; skew 500 ppm
I20260928 05:22:39.372793 23136 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:22:39 dist-test-slave-whsn krb5kdc[23123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572959, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:22:39.375671 23136 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 05:22:39.377017 23136 webserver.cc:460] Webserver started at http://127.0.97.62:35229/ using document root <none> and password file <none>
I20260928 05:22:39.377640 23136 fs_manager.cc:359] Metadata directory not provided
I20260928 05:22:39.377732 23136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:22:39.378008 23136 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:22:39.380702 23136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572622576160-388-0/minicluster-data/master-0/data/instance:
uuid: "7dc594034cd546e4969a100316d69fca"
format_stamp: "Formatted at 2026-09-28 05:22:39 on dist-test-slave-whsn"
server_key: "ec0b56f5f9bd3a80d773ded50627c9a3"
server_key_iv: "6341e29d2bbd7d29b7977f1c3a959369"
server_key_version: "encryptionkey@0"
I20260928 05:22:39.381441 23136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572622576160-388-0/minicluster-data/master-0/wal/instance:
uuid: "7dc594034cd546e4969a100316d69fca"
format_stamp: "Formatted at 2026-09-28 05:22:39 on dist-test-slave-whsn"
server_key: "ec0b56f5f9bd3a80d773ded50627c9a3"
server_key_iv: "6341e29d2bbd7d29b7977f1c3a959369"
server_key_version: "encryptionkey@0"
I20260928 05:22:39.386132 23136 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260928 05:22:39.389282 23151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:22:39.390516 23136 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.002s
I20260928 05:22:39.390671 23136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572622576160-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572622576160-388-0/minicluster-data/master-0/wal
uuid: "7dc594034cd546e4969a100316d69fca"
format_stamp: "Formatted at 2026-09-28 05:22:39 on dist-test-slave-whsn"
server_key: "ec0b56f5f9bd3a80d773ded50627c9a3"
server_key_iv: "6341e29d2bbd7d29b7977f1c3a959369"
server_key_version: "encryptionkey@0"
I20260928 05:22:39.390870 23136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572622576160-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572622576160-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572622576160-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:22:39.408414 23136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:22:39.412089 23136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:22:39.412391 23136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:22:39.429878 23136 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40853
I20260928 05:22:39.429898 23202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40853 every 8 connection(s)
I20260928 05:22:39.431198 23136 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572622576160-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:22:39.434617 23203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:22:39.440502 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 23136
I20260928 05:22:39.440662 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572622576160-388-0/minicluster-data/master-0/wal/instance
I20260928 05:22:39.441175 388 external_mini_cluster.cc:1428] Setting key c6217cdfd39710aafd59f4ff2c0de389
I20260928 05:22:39.440656 23203 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40853 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 05:22:39 dist-test-slave-whsn krb5kdc[23123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572959, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:22:39.461302 23205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:39.443677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40853 (local address 127.0.0.1:45800)
0928 05:22:39.444368 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:39.444403 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 05:22:39.446279 (+ 1876us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:22:39.447061 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:22:39.447067 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:22:39.447710 (+ 643us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:22:39.448026 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:39.448042 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:39.449523 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:39.449527 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:39.450536 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:39.450548 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:39.451392 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:39.451399 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:39.451566 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:39.452115 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:22:39.452135 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:22:39.454538 (+ 2403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:22:39.457401 (+ 2863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:22:39.457413 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:22:39.457424 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:22:39.457945 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:22:39.458306 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:22:39.458311 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:22:39.458314 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:22:39.458517 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:22:39.459174 (+ 657us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:22:39.459181 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:22:39.459366 (+ 185us) client_negotiation.cc:769] Sending connection context
0928 05:22:39.459571 (+ 205us) client_negotiation.cc:240] Negotiation successful
0928 05:22:39.460078 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"mutex_wait_us":120,"thread_start_us":269,"threads_started":1}
I20260928 05:22:39.461670 23206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:39.444838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45800 (local address 127.0.97.62:40853)
0928 05:22:39.445503 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:39.445511 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:39.445555 (+ 44us) server_negotiation.cc:407] Connection header received
0928 05:22:39.446609 (+ 1054us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:39.446615 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:39.446840 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:39.447021 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:39.448216 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:39.449392 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:39.450659 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:39.451174 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:39.451325 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:39.454750 (+ 3425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:39.454757 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:39.454766 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:22:39.454793 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:39.457278 (+ 2485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:39.458060 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:39.458066 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:39.458068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:39.458195 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:39.458615 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:39.458617 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:39.458619 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:39.459046 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:39.459222 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:39.459701 (+ 479us) server_negotiation.cc:299] Negotiation successful
0928 05:22:39.459958 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":286,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
I20260928 05:22:39.464481 23203 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40335 } has no permanent_uuid. Determining permanent_uuid...
I20260928 05:22:39.465469 23205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:39.465135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40335 (local address 127.0.0.1:47612)
0928 05:22:39.465268 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:39.465370 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
W20260928 05:22:39.465775 23154 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40335: connect: Connection refused (error 111)
I20260928 05:22:39.467365 23205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:39.466976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40335 (local address 127.0.0.1:47626)
0928 05:22:39.467122 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:39.467209 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260928 05:22:39.467566 23203 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40335: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40335: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572622576160-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572622576160-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260928 05:22:39.489900 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572622576160-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572622576160-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572622576160-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572622576160-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:40335
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572622576160-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40853,127.0.97.61:40335,127.0.97.60:45935
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 05:22:39.516870 23203 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40335 } attempt: 1
I20260928 05:22:39.517874 23205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:39.517492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40335 (local address 127.0.0.1:47638)
0928 05:22:39.517661 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:39.517755 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260928 05:22:39.518766 23205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:39.518450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40335 (local address 127.0.0.1:47644)
0928 05:22:39.518584 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:39.518645 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260928 05:22:39.518958 23203 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40335: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40335: connect: Connection refused (error 111)
I20260928 05:22:39.587186 23203 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40335 } attempt: 2
I20260928 05:22:39.588490 23205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:39.587937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40335 (local address 127.0.0.1:47654)
0928 05:22:39.588191 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:39.588300 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260928 05:22:39.589675 23205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:39.589182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40335 (local address 127.0.0.1:47668)
0928 05:22:39.589413 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:39.589513 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260928 05:22:39.589950 23203 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40335: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40335: connect: Connection refused (error 111)
I20260928 05:22:39.681198 23203 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40335 } attempt: 3
I20260928 05:22:39.682279 23205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:39.681876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40335 (local address 127.0.0.1:47670)
0928 05:22:39.682061 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:39.682150 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260928 05:22:39.683231 23205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:39.682891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40335 (local address 127.0.0.1:47682)
0928 05:22:39.683037 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:39.683102 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260928 05:22:39.683404 23203 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40335: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40335: connect: Connection refused (error 111)
W20260928 05:22:39.790655 23210 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 05:22:39.790972 23210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:22:39.791062 23210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:22:39.791170 23210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:22:39.798153 23210 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:22:39.798236 23210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:22:39.798323 23210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:22:39.798388 23210 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:22:39.798465 23210 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:22:39.807758 23210 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572622576160-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572622576160-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40853,127.0.97.61:40335,127.0.97.60:45935
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572622576160-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:40335
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572622576160-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572622576160-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:22:39.809007 23210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:22:39.810221 23210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:22:39.820713 23215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:22:39.820782 23216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:22:39.820990 23218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:22:39.822033 23210 server_base.cc:1034] running on GCE node
I20260928 05:22:39.822686 23210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:22:39.823875 23210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:22:39.825076 23210 hybrid_clock.cc:648] HybridClock initialized: now 1790572959825010 us; error 34 us; skew 500 ppm
I20260928 05:22:39.825352 23210 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 05:22:39.826599 23203 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40335 } attempt: 4
I20260928 05:22:39.827618 23205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:39.827085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40335 (local address 127.0.0.1:47684)
0928 05:22:39.827365 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:39.827459 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
Sep 28 05:22:39 dist-test-slave-whsn krb5kdc[23123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572959, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:22:39.828353 23210 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260928 05:22:39.828634 23205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:39.828266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40335 (local address 127.0.0.1:47690)
0928 05:22:39.828401 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:39.828487 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260928 05:22:39.828905 23203 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40335: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40335: connect: Connection refused (error 111)
I20260928 05:22:39.829932 23210 webserver.cc:460] Webserver started at http://127.0.97.61:35293/ using document root <none> and password file <none>
I20260928 05:22:39.830677 23210 fs_manager.cc:359] Metadata directory not provided
I20260928 05:22:39.830770 23210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:22:39.831051 23210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:22:39.833751 23210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572622576160-388-0/minicluster-data/master-1/data/instance:
uuid: "b7be9018ae7a401990bc0926ab4261ae"
format_stamp: "Formatted at 2026-09-28 05:22:39 on dist-test-slave-whsn"
server_key: "89407cc0b1782b5b301bf3e990b3a592"
server_key_iv: "f78d515015e88cabea7534a1d91927a6"
server_key_version: "encryptionkey@0"
I20260928 05:22:39.834455 23210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572622576160-388-0/minicluster-data/master-1/wal/instance:
uuid: "b7be9018ae7a401990bc0926ab4261ae"
format_stamp: "Formatted at 2026-09-28 05:22:39 on dist-test-slave-whsn"
server_key: "89407cc0b1782b5b301bf3e990b3a592"
server_key_iv: "f78d515015e88cabea7534a1d91927a6"
server_key_version: "encryptionkey@0"
I20260928 05:22:39.838953 23210 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260928 05:22:39.842093 23225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:22:39.843230 23210 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260928 05:22:39.843379 23210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572622576160-388-0/minicluster-data/master-1/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572622576160-388-0/minicluster-data/master-1/wal
uuid: "b7be9018ae7a401990bc0926ab4261ae"
format_stamp: "Formatted at 2026-09-28 05:22:39 on dist-test-slave-whsn"
server_key: "89407cc0b1782b5b301bf3e990b3a592"
server_key_iv: "f78d515015e88cabea7534a1d91927a6"
server_key_version: "encryptionkey@0"
I20260928 05:22:39.843550 23210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572622576160-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572622576160-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572622576160-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:22:39.867684 23210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:22:39.871021 23210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:22:39.871238 23210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:22:39.887297 23210 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:40335
I20260928 05:22:39.887318 23276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:40335 every 8 connection(s)
I20260928 05:22:39.888523 23210 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572622576160-388-0/minicluster-data/master-1/data/info.pb
I20260928 05:22:39.891958 23277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:22:39.896663 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 23210
I20260928 05:22:39.896816 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572622576160-388-0/minicluster-data/master-1/wal/instance
I20260928 05:22:39.897349 388 external_mini_cluster.cc:1428] Setting key a36a56ea9b5201711a31d9c3ba998fb8
I20260928 05:22:39.898226 23277 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40853 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 05:22:39 dist-test-slave-whsn krb5kdc[23123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572959, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:22:39.917143 23206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:39.900466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45810 (local address 127.0.97.62:40853)
0928 05:22:39.900618 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:39.900622 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:39.901838 (+ 1216us) server_negotiation.cc:407] Connection header received
0928 05:22:39.903750 (+ 1912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:39.903755 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:39.903834 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:39.903920 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:39.905519 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:39.906181 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:39.907645 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:39.908045 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:39.908157 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:39.911560 (+ 3403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:39.911565 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:39.911568 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:22:39.911605 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:39.914108 (+ 2503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:39.914921 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:39.914924 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:39.914926 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:39.915033 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:39.915448 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:39.915454 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:39.915456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:39.915776 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:39.915884 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:39.916699 (+ 815us) server_negotiation.cc:299] Negotiation successful
0928 05:22:39.916905 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 05:22:39.918517 23279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:39.901013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40853 (local address 127.0.0.1:45810)
0928 05:22:39.901689 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:39.901731 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 05:22:39.903497 (+ 1766us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:22:39.904163 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:22:39.904172 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:22:39.904659 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:22:39.905327 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:39.905349 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:39.906327 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:39.906331 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:39.907490 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:39.907502 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:39.908278 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:39.908285 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:39.908483 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:39.909101 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:22:39.909122 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:22:39.911331 (+ 2209us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:22:39.914243 (+ 2912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:22:39.914251 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:22:39.914266 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:22:39.914821 (+ 555us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:22:39.915120 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:22:39.915125 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:22:39.915129 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:22:39.915347 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:22:39.915896 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:22:39.915903 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:22:39.916341 (+ 438us) client_negotiation.cc:769] Sending connection context
0928 05:22:39.916643 (+ 302us) client_negotiation.cc:240] Negotiation successful
0928 05:22:39.917166 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":277,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
I20260928 05:22:39.920182 23277 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40335 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
Sep 28 05:22:39 dist-test-slave-whsn krb5kdc[23123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572959, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260928 05:22:39.933933 23279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:39.920806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40335 (local address 127.0.0.1:47700)
0928 05:22:39.920967 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:39.920982 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 05:22:39.921060 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:22:39.922092 (+ 1032us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:22:39.922097 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:22:39.922464 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:22:39.922598 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:39.922606 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:39.924386 (+ 1780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:39.924389 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:39.925240 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:39.925250 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:39.926178 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:39.926181 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:39.926298 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:39.926784 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:22:39.926798 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:22:39.928637 (+ 1839us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:22:39.931609 (+ 2972us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:22:39.931614 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:22:39.931618 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:22:39.932024 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:22:39.932342 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:22:39.932347 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:22:39.932350 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:22:39.932507 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:22:39.933064 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:22:39.933072 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:22:39.933221 (+ 149us) client_negotiation.cc:769] Sending connection context
0928 05:22:39.933349 (+ 128us) client_negotiation.cc:240] Negotiation successful
0928 05:22:39.933598 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260928 05:22:39.934016 23281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:39.921043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47700 (local address 127.0.97.61:40335)
0928 05:22:39.921643 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:39.921648 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:39.921671 (+ 23us) server_negotiation.cc:407] Connection header received
0928 05:22:39.921708 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:39.921714 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:39.921949 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:39.922078 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:39.922755 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:39.924230 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:39.925364 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:39.926051 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:39.926154 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:39.928831 (+ 2677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:39.928843 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:39.928881 (+ 38us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:22:39.928918 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:39.931490 (+ 2572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:39.932129 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:39.932134 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:39.932136 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:39.932242 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:39.932606 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:39.932609 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:39.932611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:39.932955 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:39.933110 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:39.933481 (+ 371us) server_negotiation.cc:299] Negotiation successful
0928 05:22:39.933689 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":475,"thread_start_us":350,"threads_started":1}
I20260928 05:22:39.936714 23277 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45935 } has no permanent_uuid. Determining permanent_uuid...
I20260928 05:22:39.937748 23279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:39.937277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45935 (local address 127.0.0.1:45910)
0928 05:22:39.937454 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:39.937600 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260928 05:22:39.938017 23226 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45935: connect: Connection refused (error 111)
I20260928 05:22:39.939316 23279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:39.939032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45935 (local address 127.0.0.1:45912)
0928 05:22:39.939152 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:39.939224 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260928 05:22:39.939510 23277 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45935: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45935: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572622576160-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572622576160-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260928 05:22:39.950013 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasku1htJC/build/asan/bin/kudu
/tmp/dist-test-tasku1htJC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572622576160-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572622576160-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572622576160-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572622576160-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:45935
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572622576160-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40853,127.0.97.61:40335,127.0.97.60:45935
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572622576160-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572622576160-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572622576160-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 05:22:39.988771 23277 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45935 } attempt: 1
I20260928 05:22:39.989921 23279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:39.989498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45935 (local address 127.0.0.1:45920)
0928 05:22:39.989681 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:39.989764 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260928 05:22:39.990828 23279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:39.990519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45935 (local address 127.0.0.1:45926)
0928 05:22:39.990651 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:39.990719 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260928 05:22:39.991020 23277 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45935: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45935: connect: Connection refused (error 111)
I20260928 05:22:40.059199 23277 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45935 } attempt: 2
I20260928 05:22:40.060302 23279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:40.059886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45935 (local address 127.0.0.1:45930)
0928 05:22:40.060069 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:40.060155 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260928 05:22:40.061295 23279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:40.060993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45935 (local address 127.0.0.1:45946)
0928 05:22:40.061109 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:40.061174 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260928 05:22:40.061746 23277 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45935: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45935: connect: Connection refused (error 111)
I20260928 05:22:40.128135 23203 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40335 } attempt: 5
Sep 28 05:22:40 dist-test-slave-whsn krb5kdc[23123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572959, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260928 05:22:40.141983 23205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:40.128913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40335 (local address 127.0.0.1:47710)
0928 05:22:40.129124 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:40.129145 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 05:22:40.129266 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:22:40.129628 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:22:40.129631 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:22:40.129999 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:22:40.130168 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:40.130176 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:40.131517 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:40.131520 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:40.132337 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:40.132347 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:40.133286 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:40.133290 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:40.133431 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:40.133974 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:22:40.133991 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:22:40.136075 (+ 2084us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:22:40.139746 (+ 3671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:22:40.139750 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:22:40.139752 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:22:40.140142 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:22:40.140489 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:22:40.140492 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:22:40.140495 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:22:40.140610 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:22:40.141298 (+ 688us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:22:40.141303 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:22:40.141424 (+ 121us) client_negotiation.cc:769] Sending connection context
0928 05:22:40.141534 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 05:22:40.141730 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260928 05:22:40.142239 23281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:40.129064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47710 (local address 127.0.97.61:40335)
0928 05:22:40.129292 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:40.129299 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:40.129324 (+ 25us) server_negotiation.cc:407] Connection header received
0928 05:22:40.129388 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:40.129395 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:40.129498 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:40.129614 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:40.130320 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:40.131403 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:40.132477 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:40.133163 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:40.133283 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:40.136258 (+ 2975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:40.136265 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:40.136269 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:22:40.136309 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:40.139597 (+ 3288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:40.140252 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:40.140257 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:40.140260 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:40.140398 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:40.140738 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:40.140743 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:40.140748 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:40.141190 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:40.141344 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:40.141666 (+ 322us) server_negotiation.cc:299] Negotiation successful
0928 05:22:40.141907 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 05:22:40.143231 23203 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45935 } has no permanent_uuid. Determining permanent_uuid...
I20260928 05:22:40.144209 23205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:40.143867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45935 (local address 127.0.0.1:45960)
0928 05:22:40.144016 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:40.144082 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260928 05:22:40.145166 23205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:40.144880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45935 (local address 127.0.0.1:45976)
0928 05:22:40.145003 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:40.145075 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260928 05:22:40.145368 23203 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45935: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45935: connect: Connection refused (error 111)
I20260928 05:22:40.152930 23277 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45935 } attempt: 3
I20260928 05:22:40.153682 23279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:40.153331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45935 (local address 127.0.0.1:45988)
0928 05:22:40.153491 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:40.153558 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260928 05:22:40.154536 23279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:40.154178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45935 (local address 127.0.0.1:45996)
0928 05:22:40.154372 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:40.154438 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
W20260928 05:22:40.154718 23277 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45935: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45935: connect: Connection refused (error 111)
I20260928 05:22:40.196554 23203 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45935 } attempt: 1
I20260928 05:22:40.197544 23205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:40.197146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45935 (local address 127.0.0.1:46012)
0928 05:22:40.197317 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:40.197396 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260928 05:22:40.198405 23205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:40.198099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45935 (local address 127.0.0.1:46028)
0928 05:22:40.198232 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:40.198305 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260928 05:22:40.198580 23203 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45935: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45935: connect: Connection refused (error 111)
W20260928 05:22:40.256937 23284 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 05:22:40.257241 23284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:22:40.257332 23284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:22:40.257449 23284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:22:40.264432 23284 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:22:40.264515 23284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:22:40.264601 23284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:22:40.264668 23284 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:22:40.264746 23284 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:22:40.266747 23203 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45935 } attempt: 2
I20260928 05:22:40.267752 23205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:40.267327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45935 (local address 127.0.0.1:46042)
0928 05:22:40.267525 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:40.267603 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260928 05:22:40.268805 23205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:40.268442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45935 (local address 127.0.0.1:46052)
0928 05:22:40.268612 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:40.268684 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260928 05:22:40.269058 23203 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45935: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45935: connect: Connection refused (error 111)
I20260928 05:22:40.274960 23284 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572622576160-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572622576160-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40853,127.0.97.61:40335,127.0.97.60:45935
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572622576160-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:45935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572622576160-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572622576160-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:22:40.276124 23284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:22:40.277400 23284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:22:40.287830 23290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:22:40.287787 23292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:22:40.288061 23289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:22:40.288625 23284 server_base.cc:1034] running on GCE node
I20260928 05:22:40.289278 23284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:22:40.290513 23284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:22:40.291733 23284 hybrid_clock.cc:648] HybridClock initialized: now 1790572960291674 us; error 33 us; skew 500 ppm
I20260928 05:22:40.292016 23284 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:22:40 dist-test-slave-whsn krb5kdc[23123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572960, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:22:40.294833 23284 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260928 05:22:40.296087 23284 webserver.cc:460] Webserver started at http://127.0.97.60:35337/ using document root <none> and password file <none>
I20260928 05:22:40.296715 23284 fs_manager.cc:359] Metadata directory not provided
I20260928 05:22:40.296805 23284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:22:40.297099 23284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:22:40.297883 23277 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45935 } attempt: 4
I20260928 05:22:40.298766 23279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:40.298363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45935 (local address 127.0.0.1:46066)
0928 05:22:40.298557 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:40.298630 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260928 05:22:40.299585 23279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:40.299292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45935 (local address 127.0.0.1:46076)
0928 05:22:40.299422 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:40.299486 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260928 05:22:40.299754 23277 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45935: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45935: connect: Connection refused (error 111)
I20260928 05:22:40.299832 23284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572622576160-388-0/minicluster-data/master-2/data/instance:
uuid: "17dea402d0154d9eb28b0d4dac48e00e"
format_stamp: "Formatted at 2026-09-28 05:22:40 on dist-test-slave-whsn"
server_key: "a4e51158a2c1cbd89b90fcbdb4855a19"
server_key_iv: "dff82dfc5acab3aeff485ccbaf7a749b"
server_key_version: "encryptionkey@0"
I20260928 05:22:40.300540 23284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572622576160-388-0/minicluster-data/master-2/wal/instance:
uuid: "17dea402d0154d9eb28b0d4dac48e00e"
format_stamp: "Formatted at 2026-09-28 05:22:40 on dist-test-slave-whsn"
server_key: "a4e51158a2c1cbd89b90fcbdb4855a19"
server_key_iv: "dff82dfc5acab3aeff485ccbaf7a749b"
server_key_version: "encryptionkey@0"
I20260928 05:22:40.305064 23284 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260928 05:22:40.308123 23299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:22:40.309332 23284 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 05:22:40.309496 23284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572622576160-388-0/minicluster-data/master-2/data,/tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572622576160-388-0/minicluster-data/master-2/wal
uuid: "17dea402d0154d9eb28b0d4dac48e00e"
format_stamp: "Formatted at 2026-09-28 05:22:40 on dist-test-slave-whsn"
server_key: "a4e51158a2c1cbd89b90fcbdb4855a19"
server_key_iv: "dff82dfc5acab3aeff485ccbaf7a749b"
server_key_version: "encryptionkey@0"
I20260928 05:22:40.309680 23284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572622576160-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572622576160-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572622576160-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:22:40.320304 23284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:22:40.323570 23284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:22:40.323784 23284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:22:40.339648 23284 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:45935
I20260928 05:22:40.339666 23350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:45935 every 8 connection(s)
I20260928 05:22:40.340814 23284 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572622576160-388-0/minicluster-data/master-2/data/info.pb
I20260928 05:22:40.344131 23351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:22:40.347429 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasku1htJC/build/asan/bin/kudu as pid 23284
I20260928 05:22:40.347572 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasku1htJC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572622576160-388-0/minicluster-data/master-2/wal/instance
I20260928 05:22:40.348011 388 external_mini_cluster.cc:1428] Setting key 8ecf3b7288ebe1f2b1bad6979eaf7033
I20260928 05:22:40.350965 23351 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40853 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 05:22:40 dist-test-slave-whsn krb5kdc[23123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 28 05:22:40 dist-test-slave-whsn krb5kdc[23123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572960, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:22:40.368095 23206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:40.354544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45816 (local address 127.0.97.62:40853)
0928 05:22:40.354749 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:40.354754 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:40.355089 (+ 335us) server_negotiation.cc:407] Connection header received
0928 05:22:40.355232 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:40.355235 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:40.355297 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:40.355380 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:40.356518 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:40.357203 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:40.358538 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:40.359153 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:40.359286 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:40.362296 (+ 3010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:40.362302 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:40.362307 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:22:40.362343 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:40.365398 (+ 3055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:40.366044 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:40.366047 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:40.366049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:40.366178 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:40.366506 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:40.366510 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:40.366513 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:40.366821 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:40.367007 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:40.367546 (+ 539us) server_negotiation.cc:299] Negotiation successful
0928 05:22:40.367741 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 05:22:40.368815 23354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:40.349518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46078 (local address 127.0.97.60:45935)
0928 05:22:40.350217 (+ 699us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:40.350231 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:40.350278 (+ 47us) server_negotiation.cc:407] Connection header received
0928 05:22:40.351958 (+ 1680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:40.351983 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:40.352468 (+ 485us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:40.352886 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:40.354061 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:40.355806 (+ 1745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:40.357010 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:40.357643 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:40.357812 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:40.361912 (+ 4100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:40.361919 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:40.361928 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:22:40.361959 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:40.365071 (+ 3112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:40.365675 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:40.365681 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:40.365683 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:40.365828 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:40.366194 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:40.366199 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:40.366201 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:40.366751 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:40.367074 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:40.367396 (+ 322us) server_negotiation.cc:299] Negotiation successful
0928 05:22:40.367757 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":290,"threads_started":1}
I20260928 05:22:40.368896 23355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:40.354331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40853 (local address 127.0.0.1:45816)
0928 05:22:40.354949 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:40.354992 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 05:22:40.355134 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:22:40.355449 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:22:40.355457 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:22:40.356159 (+ 702us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:22:40.356383 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:40.356392 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:40.357350 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:40.357354 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:40.358405 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:40.358419 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:40.359346 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:40.359351 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:40.359463 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:40.359955 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:22:40.359975 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:22:40.362115 (+ 2140us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:22:40.365510 (+ 3395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:22:40.365520 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:22:40.365530 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:22:40.365941 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:22:40.366281 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:22:40.366284 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:22:40.366287 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:22:40.366412 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:22:40.366961 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:22:40.366967 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:22:40.367184 (+ 217us) client_negotiation.cc:769] Sending connection context
0928 05:22:40.367435 (+ 251us) client_negotiation.cc:240] Negotiation successful
0928 05:22:40.367958 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":383,"mutex_wait_us":197,"thread_start_us":237,"threads_started":1}
I20260928 05:22:40.370522 23351 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40335 } has no permanent_uuid. Determining permanent_uuid...
I20260928 05:22:40.375264 23203 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45935 } attempt: 3
Sep 28 05:22:40 dist-test-slave-whsn krb5kdc[23123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572960, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Sep 28 05:22:40 dist-test-slave-whsn krb5kdc[23123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Sep 28 05:22:40 dist-test-slave-whsn krb5kdc[23123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572959, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260928 05:22:40.384377 23355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:40.371149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40335 (local address 127.0.0.1:47716)
0928 05:22:40.371312 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:40.371329 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 05:22:40.371409 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:22:40.372922 (+ 1513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:22:40.372925 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:22:40.373191 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:22:40.373310 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:40.373316 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:40.374321 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:40.374326 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:40.375140 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:40.375151 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:40.375973 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:40.375978 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:40.376172 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:40.376603 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:22:40.376618 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:22:40.378459 (+ 1841us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:22:40.381905 (+ 3446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:22:40.381911 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:22:40.381914 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:22:40.382360 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:22:40.382742 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:22:40.382747 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:22:40.382751 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:22:40.382907 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:22:40.383545 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:22:40.383549 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:22:40.383686 (+ 137us) client_negotiation.cc:769] Sending connection context
0928 05:22:40.383824 (+ 138us) client_negotiation.cc:240] Negotiation successful
0928 05:22:40.384069 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260928 05:22:40.384490 23281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:40.371246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47716 (local address 127.0.97.61:40335)
0928 05:22:40.371688 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:40.371694 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:40.371709 (+ 15us) server_negotiation.cc:407] Connection header received
0928 05:22:40.371762 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:40.371767 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:40.371835 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:40.371929 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:40.373443 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:40.374187 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:40.375271 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:40.375775 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:40.375884 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:40.378616 (+ 2732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:40.378620 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:40.378625 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:22:40.378661 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:40.381781 (+ 3120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:40.382472 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:40.382477 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:40.382481 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:40.382631 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:40.383012 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:40.383016 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:40.383020 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:40.383333 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:40.383483 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:40.383934 (+ 451us) server_negotiation.cc:299] Negotiation successful
0928 05:22:40.384129 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318}
I20260928 05:22:40.385839 23351 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45935 } has no permanent_uuid. Determining permanent_uuid...
I20260928 05:22:40.386075 23206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:40.371473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45828 (local address 127.0.97.62:40853)
0928 05:22:40.371617 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:40.371622 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:40.371641 (+ 19us) server_negotiation.cc:407] Connection header received
0928 05:22:40.371768 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:40.371772 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:40.371832 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:40.371925 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:40.372831 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:40.373649 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:40.374788 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:40.375180 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:40.375322 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:40.379951 (+ 4629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:40.379955 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:40.379958 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:22:40.379985 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:40.383078 (+ 3093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:40.383678 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:40.383681 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:40.383683 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:40.383816 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:40.384211 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:40.384216 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:40.384219 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:40.385048 (+ 829us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:40.385306 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:40.385592 (+ 286us) server_negotiation.cc:299] Negotiation successful
0928 05:22:40.385770 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 05:22:40.390220 23205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:40.375745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45935 (local address 127.0.0.1:46094)
0928 05:22:40.376178 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:40.376192 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 05:22:40.376268 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:22:40.376563 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:22:40.376567 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:22:40.377036 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:22:40.377215 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:40.377227 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:40.378673 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:40.378678 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:40.379909 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:40.379925 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:40.380774 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:40.380780 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:40.381032 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:40.381539 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:22:40.381562 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:22:40.384133 (+ 2571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:22:40.387669 (+ 3536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:22:40.387673 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:22:40.387677 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:22:40.388100 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:22:40.388509 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:22:40.388514 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:22:40.388518 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:22:40.388665 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:22:40.389262 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:22:40.389267 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:22:40.389461 (+ 194us) client_negotiation.cc:769] Sending connection context
0928 05:22:40.389602 (+ 141us) client_negotiation.cc:240] Negotiation successful
0928 05:22:40.389855 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331}
I20260928 05:22:40.390398 23354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:40.375946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46094 (local address 127.0.97.60:45935)
0928 05:22:40.376099 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:40.376104 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:40.376249 (+ 145us) server_negotiation.cc:407] Connection header received
0928 05:22:40.376341 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:40.376346 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:40.376408 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:40.377365 (+ 957us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:40.377502 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:40.378496 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:40.380058 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:40.380639 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:40.380754 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:40.384329 (+ 3575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:40.384335 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:40.384339 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:22:40.384386 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:40.387541 (+ 3155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:40.388244 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:40.388248 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:40.388251 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:40.388392 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:40.388777 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:40.388782 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:40.388786 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:40.389144 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:40.389272 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:40.389686 (+ 414us) server_negotiation.cc:299] Negotiation successful
0928 05:22:40.389968 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":61}
Sep 28 05:22:40 dist-test-slave-whsn krb5kdc[23123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572960, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 28 05:22:40 dist-test-slave-whsn krb5kdc[23123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260928 05:22:40.396404 23203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7dc594034cd546e4969a100316d69fca: Bootstrap starting.
I20260928 05:22:40.400104 23203 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7dc594034cd546e4969a100316d69fca: Neither blocks nor log segments found. Creating new log.
I20260928 05:22:40.400897 23356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:40.386583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46102 (local address 127.0.97.60:45935)
0928 05:22:40.387332 (+ 749us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:40.387337 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:40.387358 (+ 21us) server_negotiation.cc:407] Connection header received
0928 05:22:40.387432 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:40.387436 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:40.387499 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:40.387599 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:40.388353 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:40.389458 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:40.391045 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:40.391439 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:40.391538 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:40.394862 (+ 3324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:40.394868 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:40.394872 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:22:40.394902 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:40.397964 (+ 3062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:40.398683 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:40.398688 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:40.398692 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:40.398820 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:40.399159 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:40.399164 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:40.399167 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:40.399417 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:40.399536 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:40.400339 (+ 803us) server_negotiation.cc:299] Negotiation successful
0928 05:22:40.400525 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":470,"threads_started":1}
I20260928 05:22:40.401073 23355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:40.386404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45935 (local address 127.0.0.1:46102)
0928 05:22:40.387235 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:40.387251 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 05:22:40.387342 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:22:40.387629 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:22:40.387634 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:22:40.388055 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:22:40.388205 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:40.388215 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:40.389558 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:40.389563 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:40.390904 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:40.390916 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:40.391687 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:40.391692 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:40.391846 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:40.392335 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:22:40.392361 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:22:40.394670 (+ 2309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:22:40.398096 (+ 3426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:22:40.398101 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:22:40.398104 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:22:40.398566 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:22:40.398929 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:22:40.398934 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:22:40.398937 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:22:40.399081 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:22:40.399521 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:22:40.399525 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:22:40.399655 (+ 130us) client_negotiation.cc:769] Sending connection context
0928 05:22:40.399797 (+ 142us) client_negotiation.cc:240] Negotiation successful
0928 05:22:40.400036 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":732,"mutex_wait_us":1}
I20260928 05:22:40.401301 23203 log.cc:824] T 00000000000000000000000000000000 P 7dc594034cd546e4969a100316d69fca: Log is configured to *not* fsync() on all Append() calls
I20260928 05:22:40.402065 23281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:40.387626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47724 (local address 127.0.97.61:40335)
0928 05:22:40.387769 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:40.387774 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:40.387974 (+ 200us) server_negotiation.cc:407] Connection header received
0928 05:22:40.388089 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:40.388094 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:40.388152 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:40.388225 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:40.389508 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:40.390487 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:40.391935 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:40.392463 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:40.392569 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:40.396306 (+ 3737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:40.396312 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:40.396315 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:22:40.396342 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:40.398984 (+ 2642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:40.399655 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:40.399660 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:40.399663 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:40.399805 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:40.400187 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:40.400192 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:40.400195 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:40.400508 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:40.400760 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:40.401562 (+ 802us) server_negotiation.cc:299] Negotiation successful
0928 05:22:40.401752 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 05:22:40.404220 23203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7dc594034cd546e4969a100316d69fca: No bootstrap required, opened a new log
I20260928 05:22:40.405925 23351 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17dea402d0154d9eb28b0d4dac48e00e: Bootstrap starting.
I20260928 05:22:40.406628 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 05:22:40.409230 23351 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 17dea402d0154d9eb28b0d4dac48e00e: Neither blocks nor log segments found. Creating new log.
I20260928 05:22:40.409523 23178 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "7dc594034cd546e4969a100316d69fca"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45828
I20260928 05:22:40.410145 23351 log.cc:824] T 00000000000000000000000000000000 P 17dea402d0154d9eb28b0d4dac48e00e: Log is configured to *not* fsync() on all Append() calls
I20260928 05:22:40.412473 23351 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17dea402d0154d9eb28b0d4dac48e00e: No bootstrap required, opened a new log
I20260928 05:22:40.412739 23203 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7dc594034cd546e4969a100316d69fca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7dc594034cd546e4969a100316d69fca" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40853 } } peers { permanent_uuid: "b7be9018ae7a401990bc0926ab4261ae" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40335 } } peers { permanent_uuid: "17dea402d0154d9eb28b0d4dac48e00e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45935 } }
I20260928 05:22:40.413049 23203 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7dc594034cd546e4969a100316d69fca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7dc594034cd546e4969a100316d69fca, State: Initialized, Role: FOLLOWER
I20260928 05:22:40.413477 23203 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7dc594034cd546e4969a100316d69fca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7dc594034cd546e4969a100316d69fca" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40853 } } peers { permanent_uuid: "b7be9018ae7a401990bc0926ab4261ae" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40335 } } peers { permanent_uuid: "17dea402d0154d9eb28b0d4dac48e00e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45935 } }
I20260928 05:22:40.414572 23359 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7dc594034cd546e4969a100316d69fca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7dc594034cd546e4969a100316d69fca" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40853 } } peers { permanent_uuid: "b7be9018ae7a401990bc0926ab4261ae" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40335 } } peers { permanent_uuid: "17dea402d0154d9eb28b0d4dac48e00e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45935 } } }
I20260928 05:22:40.415212 23359 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7dc594034cd546e4969a100316d69fca [sys.catalog]: This master's current role is: FOLLOWER
I20260928 05:22:40.415341 23203 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7dc594034cd546e4969a100316d69fca [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:22:40.415412 23178 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7dc594034cd546e4969a100316d69fca [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 05:22:40.415683 23178 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7dc594034cd546e4969a100316d69fca [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:22:40.419135 23178 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7dc594034cd546e4969a100316d69fca [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7dc594034cd546e4969a100316d69fca" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40853 } } peers { permanent_uuid: "b7be9018ae7a401990bc0926ab4261ae" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40335 } } peers { permanent_uuid: "17dea402d0154d9eb28b0d4dac48e00e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45935 } }
I20260928 05:22:40.420442 23351 raft_consensus.cc:348] T 00000000000000000000000000000000 P 17dea402d0154d9eb28b0d4dac48e00e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7dc594034cd546e4969a100316d69fca" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40853 } } peers { permanent_uuid: "b7be9018ae7a401990bc0926ab4261ae" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40335 } } peers { permanent_uuid: "17dea402d0154d9eb28b0d4dac48e00e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45935 } }
I20260928 05:22:40.420816 23351 raft_consensus.cc:729] T 00000000000000000000000000000000 P 17dea402d0154d9eb28b0d4dac48e00e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17dea402d0154d9eb28b0d4dac48e00e, State: Initialized, Role: FOLLOWER
I20260928 05:22:40.421284 23351 consensus_queue.cc:260] T 00000000000000000000000000000000 P 17dea402d0154d9eb28b0d4dac48e00e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7dc594034cd546e4969a100316d69fca" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40853 } } peers { permanent_uuid: "b7be9018ae7a401990bc0926ab4261ae" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40335 } } peers { permanent_uuid: "17dea402d0154d9eb28b0d4dac48e00e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45935 } }
I20260928 05:22:40.421445 23178 leader_election.cc:288] T 00000000000000000000000000000000 P 7dc594034cd546e4969a100316d69fca [CANDIDATE]: Term 1 election: Requested vote from peers b7be9018ae7a401990bc0926ab4261ae (127.0.97.61:40335), 17dea402d0154d9eb28b0d4dac48e00e (127.0.97.60:45935)
I20260928 05:22:40.428689 23364 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17dea402d0154d9eb28b0d4dac48e00e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7dc594034cd546e4969a100316d69fca" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40853 } } peers { permanent_uuid: "b7be9018ae7a401990bc0926ab4261ae" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40335 } } peers { permanent_uuid: "17dea402d0154d9eb28b0d4dac48e00e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45935 } } }
I20260928 05:22:40.429857 23351 sys_catalog.cc:564] T 00000000000000000000000000000000 P 17dea402d0154d9eb28b0d4dac48e00e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:22:40.431224 23364 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17dea402d0154d9eb28b0d4dac48e00e [sys.catalog]: This master's current role is: FOLLOWER
I20260928 05:22:40.433049 23326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7dc594034cd546e4969a100316d69fca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "17dea402d0154d9eb28b0d4dac48e00e"
I20260928 05:22:40.433516 23326 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 17dea402d0154d9eb28b0d4dac48e00e [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:22:40.436841 23326 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 17dea402d0154d9eb28b0d4dac48e00e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7dc594034cd546e4969a100316d69fca in term 1.
I20260928 05:22:40.437487 23152 leader_election.cc:302] T 00000000000000000000000000000000 P 7dc594034cd546e4969a100316d69fca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17dea402d0154d9eb28b0d4dac48e00e, 7dc594034cd546e4969a100316d69fca; no voters:
I20260928 05:22:40.437906 23359 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7dc594034cd546e4969a100316d69fca [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:22:40.459028 23359 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7dc594034cd546e4969a100316d69fca [term 1 LEADER]: Becoming Leader. State: Replica: 7dc594034cd546e4969a100316d69fca, State: Running, Role: LEADER
I20260928 05:22:40.459987 23359 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7dc594034cd546e4969a100316d69fca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7dc594034cd546e4969a100316d69fca" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40853 } } peers { permanent_uuid: "b7be9018ae7a401990bc0926ab4261ae" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40335 } } peers { permanent_uuid: "17dea402d0154d9eb28b0d4dac48e00e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45935 } }
I20260928 05:22:40.429286 23136 master_runner.cc:186] Error getting master registration for 127.0.97.61:40335: OK, instance_id { permanent_uuid: "b7be9018ae7a401990bc0926ab4261ae" instance_seqno: 1790572959882807 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 05:22:40.482434 23136 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 05:22:40.485423 23378 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7dc594034cd546e4969a100316d69fca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7dc594034cd546e4969a100316d69fca. Latest consensus state: current_term: 1 leader_uuid: "7dc594034cd546e4969a100316d69fca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7dc594034cd546e4969a100316d69fca" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40853 } } peers { permanent_uuid: "b7be9018ae7a401990bc0926ab4261ae" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40335 } } peers { permanent_uuid: "17dea402d0154d9eb28b0d4dac48e00e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45935 } } }
I20260928 05:22:40.485886 23378 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7dc594034cd546e4969a100316d69fca [sys.catalog]: This master's current role is: LEADER
I20260928 05:22:40.487464 23206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:40.443039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45834 (local address 127.0.97.62:40853)
0928 05:22:40.443192 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:40.443199 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:40.458279 (+ 15080us) server_negotiation.cc:407] Connection header received
0928 05:22:40.460643 (+ 2364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:40.460648 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:40.460729 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:40.460812 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:40.466994 (+ 6182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:40.468093 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:40.473352 (+ 5259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:40.474028 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:40.474117 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:40.480292 (+ 6175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:40.480300 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:40.480306 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:22:40.480356 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:40.484164 (+ 3808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:40.484882 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:40.484889 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:40.484893 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:40.485096 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:40.485468 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:40.485473 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:40.485476 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:40.485847 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:40.486255 (+ 408us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:40.486842 (+ 587us) server_negotiation.cc:299] Negotiation successful
0928 05:22:40.487083 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260928 05:22:40.491366 23373 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7dc594034cd546e4969a100316d69fca: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 05:22:40.491483 23373 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 05:22:40.496079 23281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:40.442755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47734 (local address 127.0.97.61:40335)
0928 05:22:40.442943 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:40.442949 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:40.464398 (+ 21449us) server_negotiation.cc:407] Connection header received
0928 05:22:40.464754 (+ 356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:40.464759 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:40.464877 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:40.464943 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:40.470087 (+ 5144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:40.471197 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:40.477997 (+ 6800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:40.478626 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:40.478714 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:40.485416 (+ 6702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:40.485422 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:40.485432 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:22:40.485474 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:40.488944 (+ 3470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:40.492262 (+ 3318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:40.492268 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:40.492272 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:40.492426 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:40.494366 (+ 1940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:40.494372 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:40.494375 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:40.494730 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:40.494971 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:40.495495 (+ 524us) server_negotiation.cc:299] Negotiation successful
0928 05:22:40.495725 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 05:22:40.504959 23356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:40.458759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46114 (local address 127.0.97.60:45935)
0928 05:22:40.458934 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:40.458941 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:40.475405 (+ 16464us) server_negotiation.cc:407] Connection header received
0928 05:22:40.476077 (+ 672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:40.476082 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:40.476179 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:40.476283 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:40.477811 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:40.478933 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:40.485902 (+ 6969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:40.486506 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:40.486598 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:40.494166 (+ 7568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:40.494173 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:40.494179 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:22:40.494220 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:40.498127 (+ 3907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:40.499471 (+ 1344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:40.499475 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:40.499478 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:40.499844 (+ 366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:40.500376 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:40.500381 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:40.500385 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:40.501524 (+ 1139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:40.504214 (+ 2690us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:40.504359 (+ 145us) server_negotiation.cc:299] Negotiation successful
0928 05:22:40.504589 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 05:22:40.509438 23390 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:22:40.511438 23390 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:22:40.510139 23284 master_runner.cc:186] Error getting master registration for 127.0.97.61:40335: OK, instance_id { permanent_uuid: "b7be9018ae7a401990bc0926ab4261ae" instance_seqno: 1790572959882807 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 05:22:40.519677 23284 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260928 05:22:40.525888 23393 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 17dea402d0154d9eb28b0d4dac48e00e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 05:22:40.526013 23393 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 05:22:40.527799 23326 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 17dea402d0154d9eb28b0d4dac48e00e [term 1 FOLLOWER]: Refusing update from remote peer 7dc594034cd546e4969a100316d69fca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:22:40.528676 23378 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7dc594034cd546e4969a100316d69fca [LEADER]: Connected to new peer: Peer: permanent_uuid: "17dea402d0154d9eb28b0d4dac48e00e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:22:40.542989 23364 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17dea402d0154d9eb28b0d4dac48e00e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7dc594034cd546e4969a100316d69fca. Latest consensus state: current_term: 1 leader_uuid: "7dc594034cd546e4969a100316d69fca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7dc594034cd546e4969a100316d69fca" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40853 } } peers { permanent_uuid: "b7be9018ae7a401990bc0926ab4261ae" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40335 } } peers { permanent_uuid: "17dea402d0154d9eb28b0d4dac48e00e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45935 } } }
I20260928 05:22:40.543273 23364 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17dea402d0154d9eb28b0d4dac48e00e [sys.catalog]: This master's current role is: FOLLOWER
I20260928 05:22:40.546388 23359 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7dc594034cd546e4969a100316d69fca [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7dc594034cd546e4969a100316d69fca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7dc594034cd546e4969a100316d69fca" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40853 } } peers { permanent_uuid: "b7be9018ae7a401990bc0926ab4261ae" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40335 } } peers { permanent_uuid: "17dea402d0154d9eb28b0d4dac48e00e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45935 } } }
I20260928 05:22:40.546710 23359 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7dc594034cd546e4969a100316d69fca [sys.catalog]: This master's current role is: LEADER
I20260928 05:22:40.552724 23390 catalog_manager.cc:1347] Generated new cluster ID: 88387a526b21496aab622ae31901e357
I20260928 05:22:40.552824 23390 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:22:40.561299 23364 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17dea402d0154d9eb28b0d4dac48e00e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7dc594034cd546e4969a100316d69fca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7dc594034cd546e4969a100316d69fca" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40853 } } peers { permanent_uuid: "b7be9018ae7a401990bc0926ab4261ae" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40335 } } peers { permanent_uuid: "17dea402d0154d9eb28b0d4dac48e00e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45935 } } }
I20260928 05:22:40.561529 23364 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17dea402d0154d9eb28b0d4dac48e00e [sys.catalog]: This master's current role is: FOLLOWER
I20260928 05:22:40.575134 23390 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:22:40.577577 23390 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:22:40.593840 23390 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7dc594034cd546e4969a100316d69fca: Generated new TSK 0
I20260928 05:22:40.595479 23390 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 05:22:40.598945 23277 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45935 } attempt: 5
Sep 28 05:22:40 dist-test-slave-whsn krb5kdc[23123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572959, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260928 05:22:40.618629 23279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:40.599430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45935 (local address 127.0.0.1:46128)
0928 05:22:40.599655 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:40.599674 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 05:22:40.599773 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:22:40.600062 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:22:40.600066 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:22:40.600464 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:22:40.600610 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:40.600618 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:40.602006 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:40.602011 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:40.602895 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:40.602905 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:40.604043 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:40.604049 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:40.604231 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:40.604880 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:22:40.604906 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:22:40.610735 (+ 5829us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:22:40.614837 (+ 4102us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:22:40.614841 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:22:40.614845 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:22:40.615338 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:22:40.616243 (+ 905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:22:40.616260 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:22:40.616264 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:22:40.616732 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:22:40.617736 (+ 1004us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:22:40.617741 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:22:40.617918 (+ 177us) client_negotiation.cc:769] Sending connection context
0928 05:22:40.618035 (+ 117us) client_negotiation.cc:240] Negotiation successful
0928 05:22:40.618278 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117}
I20260928 05:22:40.620147 23356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:40.599599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46128 (local address 127.0.97.60:45935)
0928 05:22:40.599738 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:40.599743 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:40.599760 (+ 17us) server_negotiation.cc:407] Connection header received
0928 05:22:40.599874 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:40.599878 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:40.599945 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:40.600023 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:40.600747 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:40.601889 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:40.603022 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:40.603711 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:40.603824 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:40.610963 (+ 7139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:40.610970 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:40.610975 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:22:40.611017 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:40.614640 (+ 3623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:40.615512 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:40.615521 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:40.615525 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:40.616075 (+ 550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:40.616972 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:40.616977 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:40.616981 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:40.617568 (+ 587us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:40.619100 (+ 1532us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:40.619284 (+ 184us) server_negotiation.cc:299] Negotiation successful
0928 05:22:40.619617 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 05:22:40.625706 23277 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7be9018ae7a401990bc0926ab4261ae: Bootstrap starting.
I20260928 05:22:40.638509 23277 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b7be9018ae7a401990bc0926ab4261ae: Neither blocks nor log segments found. Creating new log.
I20260928 05:22:40.639747 23277 log.cc:824] T 00000000000000000000000000000000 P b7be9018ae7a401990bc0926ab4261ae: Log is configured to *not* fsync() on all Append() calls
I20260928 05:22:40.642849 23277 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7be9018ae7a401990bc0926ab4261ae: No bootstrap required, opened a new log
I20260928 05:22:40.654453 23277 raft_consensus.cc:348] T 00000000000000000000000000000000 P b7be9018ae7a401990bc0926ab4261ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7dc594034cd546e4969a100316d69fca" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40853 } } peers { permanent_uuid: "b7be9018ae7a401990bc0926ab4261ae" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40335 } } peers { permanent_uuid: "17dea402d0154d9eb28b0d4dac48e00e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45935 } }
I20260928 05:22:40.654947 23277 raft_consensus.cc:729] T 00000000000000000000000000000000 P b7be9018ae7a401990bc0926ab4261ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7be9018ae7a401990bc0926ab4261ae, State: Initialized, Role: FOLLOWER
I20260928 05:22:40.655503 23277 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7be9018ae7a401990bc0926ab4261ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7dc594034cd546e4969a100316d69fca" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40853 } } peers { permanent_uuid: "b7be9018ae7a401990bc0926ab4261ae" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40335 } } peers { permanent_uuid: "17dea402d0154d9eb28b0d4dac48e00e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45935 } }
I20260928 05:22:40.669386 23402 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7be9018ae7a401990bc0926ab4261ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7dc594034cd546e4969a100316d69fca" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40853 } } peers { permanent_uuid: "b7be9018ae7a401990bc0926ab4261ae" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40335 } } peers { permanent_uuid: "17dea402d0154d9eb28b0d4dac48e00e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45935 } } }
I20260928 05:22:40.669813 23402 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7be9018ae7a401990bc0926ab4261ae [sys.catalog]: This master's current role is: FOLLOWER
I20260928 05:22:40.670887 23277 sys_catalog.cc:564] T 00000000000000000000000000000000 P b7be9018ae7a401990bc0926ab4261ae [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:22:40.671272 23251 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b7be9018ae7a401990bc0926ab4261ae [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:22:40.675094 23251 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b7be9018ae7a401990bc0926ab4261ae [term 1 FOLLOWER]: Refusing update from remote peer 7dc594034cd546e4969a100316d69fca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:22:40.675832 23252 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7dc594034cd546e4969a100316d69fca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b7be9018ae7a401990bc0926ab4261ae"
I20260928 05:22:40.679368 23252 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b7be9018ae7a401990bc0926ab4261ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7dc594034cd546e4969a100316d69fca in term 1.
I20260928 05:22:40.680501 23378 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7dc594034cd546e4969a100316d69fca [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7be9018ae7a401990bc0926ab4261ae" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:22:40.726698 23402 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7be9018ae7a401990bc0926ab4261ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7dc594034cd546e4969a100316d69fca. Latest consensus state: current_term: 1 leader_uuid: "7dc594034cd546e4969a100316d69fca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7dc594034cd546e4969a100316d69fca" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40853 } } peers { permanent_uuid: "b7be9018ae7a401990bc0926ab4261ae" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40335 } } peers { permanent_uuid: "17dea402d0154d9eb28b0d4dac48e00e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45935 } } }
I20260928 05:22:40.727066 23402 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7be9018ae7a401990bc0926ab4261ae [sys.catalog]: This master's current role is: FOLLOWER
I20260928 05:22:40.726838 23414 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7be9018ae7a401990bc0926ab4261ae [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7dc594034cd546e4969a100316d69fca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7dc594034cd546e4969a100316d69fca" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40853 } } peers { permanent_uuid: "b7be9018ae7a401990bc0926ab4261ae" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40335 } } peers { permanent_uuid: "17dea402d0154d9eb28b0d4dac48e00e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45935 } } }
I20260928 05:22:40.727248 23414 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7be9018ae7a401990bc0926ab4261ae [sys.catalog]: This master's current role is: FOLLOWER
I20260928 05:22:40.745517 23423 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260928 05:22:40.756049 23206 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:22:40.747065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45842 (local address 127.0.97.62:40853)
0928 05:22:40.747477 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:40.747482 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:40.747500 (+ 18us) server_negotiation.cc:407] Connection header received
0928 05:22:40.747548 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:40.747553 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:40.747639 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:40.747731 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:40.755755 (+ 8024us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45842: BlockingRecv error: recv got EOF from 127.0.0.1:45842 (error 108)
Metrics: {"server-negotiator.queue_time_us":278}
W20260928 05:22:40.758103 23281 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:22:40.748480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47750 (local address 127.0.97.61:40335)
0928 05:22:40.748622 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:40.748627 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:40.756618 (+ 7991us) server_negotiation.cc:407] Connection header received
0928 05:22:40.756878 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:40.756884 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:40.756958 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:40.757026 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:40.757901 (+ 875us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47750: BlockingRecv error: recv got EOF from 127.0.0.1:47750 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260928 05:22:40.760082 23356 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:22:40.750073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46144 (local address 127.0.97.60:45935)
0928 05:22:40.750239 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:40.750244 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:40.758659 (+ 8415us) server_negotiation.cc:407] Connection header received
0928 05:22:40.758875 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:40.758880 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:40.758951 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:40.759015 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:40.759838 (+ 823us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46144: BlockingRecv error: recv got EOF from 127.0.0.1:46144 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260928 05:22:40.770445 23281 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:22:40.764813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47762 (local address 127.0.97.61:40335)
0928 05:22:40.765019 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:40.765025 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:40.769319 (+ 4294us) server_negotiation.cc:407] Connection header received
0928 05:22:40.769484 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:40.769487 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:40.769573 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:40.769652 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:40.770267 (+ 615us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47762: BlockingRecv error: recv got EOF from 127.0.0.1:47762 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
W20260928 05:22:40.776391 23281 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:22:40.774813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47768 (local address 127.0.97.61:40335)
0928 05:22:40.774958 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:40.774963 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:40.775369 (+ 406us) server_negotiation.cc:407] Connection header received
0928 05:22:40.775526 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:40.775532 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:40.775595 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:40.775669 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:40.776227 (+ 558us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47768: BlockingRecv error: recv got EOF from 127.0.0.1:47768 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260928 05:22:40.782454 23281 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:22:40.780782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47770 (local address 127.0.97.61:40335)
0928 05:22:40.780962 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:40.780968 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:40.781377 (+ 409us) server_negotiation.cc:407] Connection header received
0928 05:22:40.781530 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:40.781535 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:40.781605 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:40.781682 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:40.782250 (+ 568us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47770: BlockingRecv error: recv got EOF from 127.0.0.1:47770 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260928 05:22:40.790377 23281 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:22:40.788739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47784 (local address 127.0.97.61:40335)
0928 05:22:40.788949 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:40.788955 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:40.789351 (+ 396us) server_negotiation.cc:407] Connection header received
0928 05:22:40.789505 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:40.789511 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:40.789573 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:40.789648 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:40.790198 (+ 550us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47784: BlockingRecv error: recv got EOF from 127.0.0.1:47784 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
W20260928 05:22:40.802307 23281 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:22:40.800712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47796 (local address 127.0.97.61:40335)
0928 05:22:40.800892 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:40.800899 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:40.801303 (+ 404us) server_negotiation.cc:407] Connection header received
0928 05:22:40.801449 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:40.801454 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:40.801506 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:40.801580 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:40.802133 (+ 553us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47796: BlockingRecv error: recv got EOF from 127.0.0.1:47796 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260928 05:22:40.822542 23281 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:22:40.820790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47806 (local address 127.0.97.61:40335)
0928 05:22:40.820959 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:40.820965 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:40.821376 (+ 411us) server_negotiation.cc:407] Connection header received
0928 05:22:40.821537 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:40.821541 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:40.821600 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:40.821676 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:40.822315 (+ 639us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47806: BlockingRecv error: recv got EOF from 127.0.0.1:47806 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260928 05:22:40.859254 23281 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:22:40.857312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47812 (local address 127.0.97.61:40335)
0928 05:22:40.857505 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:40.857512 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:40.857986 (+ 474us) server_negotiation.cc:407] Connection header received
0928 05:22:40.858176 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:40.858180 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:40.858248 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:40.858330 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:40.859025 (+ 695us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47812: BlockingRecv error: recv got EOF from 127.0.0.1:47812 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260928 05:22:40.928143 23281 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:22:40.926157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47820 (local address 127.0.97.61:40335)
0928 05:22:40.926393 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:40.926399 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:40.926838 (+ 439us) server_negotiation.cc:407] Connection header received
0928 05:22:40.927061 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:40.927065 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:40.927136 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:40.927222 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:40.927944 (+ 722us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47820: BlockingRecv error: recv got EOF from 127.0.0.1:47820 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260928 05:22:41.063107 23281 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:22:41.061145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47828 (local address 127.0.97.61:40335)
0928 05:22:41.061382 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:41.061390 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:41.061706 (+ 316us) server_negotiation.cc:407] Connection header received
0928 05:22:41.061918 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:41.061923 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:41.062002 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:41.062097 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:41.062890 (+ 793us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47828: BlockingRecv error: recv got EOF from 127.0.0.1:47828 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 05:22:41.297037 23378 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7dc594034cd546e4969a100316d69fca [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7dc594034cd546e4969a100316d69fca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7dc594034cd546e4969a100316d69fca" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40853 } } peers { permanent_uuid: "b7be9018ae7a401990bc0926ab4261ae" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40335 } } peers { permanent_uuid: "17dea402d0154d9eb28b0d4dac48e00e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45935 } } }
I20260928 05:22:41.297319 23378 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7dc594034cd546e4969a100316d69fca [sys.catalog]: This master's current role is: LEADER
W20260928 05:22:41.323920 23281 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:22:41.322280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47834 (local address 127.0.97.61:40335)
0928 05:22:41.322443 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:41.322448 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:41.322698 (+ 250us) server_negotiation.cc:407] Connection header received
0928 05:22:41.322882 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:41.322886 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:41.322956 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:41.323049 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:41.323706 (+ 657us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47834: BlockingRecv error: recv got EOF from 127.0.0.1:47834 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 05:22:41.527889 23393 catalog_manager.cc:1259] Loaded cluster ID: 88387a526b21496aab622ae31901e357
I20260928 05:22:41.527993 23393 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 17dea402d0154d9eb28b0d4dac48e00e: loading cluster ID for follower catalog manager: success
I20260928 05:22:41.530365 23393 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 17dea402d0154d9eb28b0d4dac48e00e: acquiring CA information for follower catalog manager: success
I20260928 05:22:41.532099 23393 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 17dea402d0154d9eb28b0d4dac48e00e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 05:22:41.749743 23423 catalog_manager.cc:1259] Loaded cluster ID: 88387a526b21496aab622ae31901e357
I20260928 05:22:41.749874 23423 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b7be9018ae7a401990bc0926ab4261ae: loading cluster ID for follower catalog manager: success
I20260928 05:22:41.752351 23423 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b7be9018ae7a401990bc0926ab4261ae: acquiring CA information for follower catalog manager: success
I20260928 05:22:41.754103 23423 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b7be9018ae7a401990bc0926ab4261ae: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 05:22:41.853820 23486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:41.847804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47848 (local address 127.0.97.61:40335)
0928 05:22:41.848627 (+ 823us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:41.848633 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:41.848650 (+ 17us) server_negotiation.cc:407] Connection header received
0928 05:22:41.848716 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:41.848720 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:41.848901 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:41.849025 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:22:41.849468 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:41.850291 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:41.851579 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:41.851986 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:41.852097 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:41.852582 (+ 485us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:22:41.853205 (+ 623us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:22:41.853339 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:41.853468 (+ 129us) server_negotiation.cc:299] Negotiation successful
0928 05:22:41.853545 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":693,"thread_start_us":550,"threads_started":1}
I20260928 05:22:41.865576 23492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:41.859643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46154 (local address 127.0.97.60:45935)
0928 05:22:41.860200 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:41.860205 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:41.860220 (+ 15us) server_negotiation.cc:407] Connection header received
0928 05:22:41.860303 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:41.860307 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:41.860459 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:41.860583 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:22:41.861030 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:41.861692 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:41.863494 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:41.863849 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:41.863936 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:41.864453 (+ 517us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:22:41.864955 (+ 502us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:22:41.865086 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:41.865234 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 05:22:41.865309 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":346,"threads_started":1}
I20260928 05:22:41.888918 23486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:41.871636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47858 (local address 127.0.97.61:40335)
0928 05:22:41.877437 (+ 5801us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:41.877445 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:41.881580 (+ 4135us) server_negotiation.cc:407] Connection header received
0928 05:22:41.881830 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:41.881835 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:41.881915 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:41.881974 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:22:41.882755 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:41.883857 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:41.886744 (+ 2887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:41.887398 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:41.887487 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:41.888115 (+ 628us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:22:41.888236 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:22:41.888296 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:41.888521 (+ 225us) server_negotiation.cc:299] Negotiation successful
0928 05:22:41.888596 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5662}
I20260928 05:22:41.891743 23492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:41.881792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46162 (local address 127.0.97.60:45935)
0928 05:22:41.881945 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:41.881951 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:41.885246 (+ 3295us) server_negotiation.cc:407] Connection header received
0928 05:22:41.885378 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:41.885384 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:41.885480 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:41.885572 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:22:41.886366 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:41.887416 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:41.889617 (+ 2201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:41.890263 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:41.890354 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:41.890990 (+ 636us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:22:41.891111 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:22:41.891170 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:41.891389 (+ 219us) server_negotiation.cc:299] Negotiation successful
0928 05:22:41.891462 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 05:22:41.892688 23499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:41.871275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45850 (local address 127.0.97.62:40853)
0928 05:22:41.881302 (+ 10027us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:41.881309 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:41.881330 (+ 21us) server_negotiation.cc:407] Connection header received
0928 05:22:41.881447 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:41.881454 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:41.881653 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:41.882495 (+ 842us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:22:41.882685 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:41.883886 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:41.886291 (+ 2405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:41.886943 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:41.887056 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:41.891393 (+ 4337us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:22:41.891992 (+ 599us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:22:41.892079 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:41.892295 (+ 216us) server_negotiation.cc:299] Negotiation successful
0928 05:22:41.892376 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9873,"thread_start_us":9744,"threads_started":1}
I20260928 05:22:41.898916 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 23136
I20260928 05:22:41.899931 23198 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:22:42.006505 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 23136
I20260928 05:22:42.023983 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 23210
I20260928 05:22:42.025259 23272 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:22:42.136483 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 23210
I20260928 05:22:42.151813 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasku1htJC/build/asan/bin/kudu pid 23284
I20260928 05:22:42.152782 23346 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:22:42.255578 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasku1htJC/build/asan/bin/kudu with pid 23284
2026-09-28T05:22:42Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4456 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (8975 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (339643 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST